Re: [petsc-dev] Webpage missing

[Smith, Barry F.]

   I think because there is only one way to do it and it is straightforward now 
the document was removed. You should check that the users manual does a good 
job explaining it.

   Barry


> On Mar 2, 2018, at 2:09 PM, Matthew Knepley  wrote:
> 
> I cannot find
> 
>   
> http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Sys/UsingFortran.html
> 
> in the dev docs.
> 
>   Thanks,
> 
> Matt
> 
> -- 
> What most experimenters take for granted before they begin their experiments 
> is infinitely more interesting than any results to which their experiments 
> lead.
> -- Norbert Wiener
> 
> https://www.cse.buffalo.edu/~knepley/

Re: [petsc-dev] plans for preconditioners for SeqSELL

[Smith, Barry F.]

> On Mar 4, 2018, at 6:16 PM, Jed Brown  wrote:
> 
> Matthew Knepley  writes:
> 
>> On Sun, Mar 4, 2018 at 6:57 PM, Jed Brown  wrote:
>> 
>>> "Zhang, Hong"  writes:
>>> 
 We have thought about subclassing long before. As Barry suggested, an
 ideal solution is to make a base class (XAIJ ?) for AIJ, BAIJ, SBAIJ,
 and SELL. The base class will have diagonal and off diagonal parts
 without binding their storage format to CSR-type. This requires a lot
 of efforts to refactor the current AIJ. But I think we should
 eventually do it.
>>> 
>>> It sounds like we're mixing the concerns of parallel decomposition (for
>>> which "Seq*" isn't relevant) with the sequential format as some CSR-like
>>> variant.  And apart from a couple abstraction-breaking performance
>>> optimizations (that may not be necessary and could be done differently),
>>> they are nearly independent.
>>> 
>>> Now there are some good reasons why we might want SeqAIJ to run inside a
>>> different parallel decomposition.  MATIS is one such format.  If we're
>>> serious about working with irregular problems, we'll eventually need a
>>> 2D sparse decomposition.  We could get it from an existing package such
>>> as CombBLAS, but if we do preconditioners that depend on matrix format,
>>> we'll likely want it to work on some native format.
>>> 
>>> Also, we need a better way to do small-size specialization.  I
>>> personally like doing it via inline functions instead of macros and
>>> copy-pasta, though the XL compiler can be offensively negligent.
>>> 
>> 
>> I have missed the larger point. What do you think should be done?
> 
> I think the stuff that is currently called "MPIAIJ" should be refactored
> to not depend on AIJ at all.  That is kind of necessary anyway if SELL
> is going to be made to derive from AIJ without also maintaining
> AIJ-format storage.  Once refactored, that stuff that is currentnly
> called "MPIAJ" should be renamed to something that does not include the
> substring "AIJ".

   I think this is the plan.
> 
> Or are you asking about specialization?

Re: [petsc-dev] plans for preconditioners for SeqSELL

[Smith, Barry F.]

> On Mar 4, 2018, at 6:18 PM, Jed Brown  wrote:
> 
> "Smith, Barry F."  writes:
> 
>>> I think the stuff that is currently called "MPIAIJ" should be refactored
>>> to not depend on AIJ at all.  That is kind of necessary anyway if SELL
>>> is going to be made to derive from AIJ without also maintaining
>>> AIJ-format storage.  Once refactored, that stuff that is currentnly
>>> called "MPIAJ" should be renamed to something that does not include the
>>> substring "AIJ".
>> 
>>   I think this is the plan.
> 
> Great!

  This is large project and probably not a priority. Great to have the new code 
but most other new functionality would have a higher priority.

  Barry

Re: [petsc-dev] PETSc blame digest (next) 2018-03-05

[Smith, Barry F.]

   Fixed in master and next (it was actually fixed already in master but did 
not propagate up to next as quickly as it should have).


> On Mar 5, 2018, at 9:10 AM, Jed Brown  wrote:
> 
> These warnings are actually attributable to this commit.
> 
> commit 153a8027956d231343955b351a2c77b7b409abeb
> Author: Barry Smith 
> Date:   Fri Mar 2 20:16:55 2018 -0600
> 
>Redefine how the n in PetscStrncat() works to make it more useful and not 
> require tracking the current string size
> 
>Commit-type: feature
> 
> 
> Barry, with PetscStrncpy() actually reflecting strlcpy() semantics,
> should it also be renamed?
> 
> PETSc checkBuilds  writes:
> 
>> Dear PETSc developer,
>> 
>> This email contains listings of contributions attributed to you by
>> `git blame` that caused compiler errors or warnings in PETSc automated
>> testing.  Follow the links to see the full log files. Please attempt to fix
>> the issues promptly or let us know at petsc-dev@mcs.anl.gov if you are unable
>> to resolve the issues.
>> 
>> Thanks,
>>  The PETSc development team
>> 
>> 
>> 
>> warnings attributed to commit 
>> https://bitbucket.org/petsc/petsc/commits/13f021e
>> Fix some buffer overflows for getting a display.
>> 
>>  src/sys/utils/pdisplay.c:136
>>
>> [http://ftp.mcs.anl.gov/pub/petsc/nightlylogs/archive/2018/03/05/build_next_arch-linux-cmplx-single_steamroller.log]
>>  /sandbox/petsc/petsc.next-3/src/sys/utils/pdisplay.c:136:12: warning: 
>> unused variable 'len' [-Wunused-variable]
>>
>> [http://ftp.mcs.anl.gov/pub/petsc/nightlylogs/archive/2018/03/05/build_next_arch-linux-dbg-quad_churn.log]
>>  /sandbox/petsc/petsc.next-3/src/sys/utils/pdisplay.c:136:12: warning: 
>> unused variable 'len' [-Wunused-variable]
>>
>> [http://ftp.mcs.anl.gov/pub/petsc/nightlylogs/archive/2018/03/05/build_next_arch-linux-gcov_cg.log]
>>  /sandbox/petsc/petsc.next-3/src/sys/utils/pdisplay.c:136:12: warning: 
>> unused variable 'len' [-Wunused-variable]
>>
>> [http://ftp.mcs.anl.gov/pub/petsc/nightlylogs/archive/2018/03/05/build_next_arch-linux-pkgs-cxx-mlib_el6.log]
>>  /home/sandbox/petsc/petsc.next-3/src/sys/utils/pdisplay.c:136:12: 
>> warning: unused variable 'len' [-Wunused-variable]
>>
>> [http://ftp.mcs.anl.gov/pub/petsc/nightlylogs/archive/2018/03/05/build_next_arch-freebsd-pkgs-opt_wii.log]
>>  /usr/home/balay/petsc.next-3/src/sys/utils/pdisplay.c:136:12: warning: 
>> unused variable 'len' [-Wunused-variable]
>>
>> [http://ftp.mcs.anl.gov/pub/petsc/nightlylogs/archive/2018/03/05/build_next_arch-linux-pkgs-64idx_thrash.log]
>>  /sandbox/petsc/petsc.next-3/src/sys/utils/pdisplay.c:136:12: warning: 
>> unused variable 'len' [-Wunused-variable]
>>
>> [http://ftp.mcs.anl.gov/pub/petsc/nightlylogs/archive/2018/03/05/build_next_arch-osx-10.6-cxx-cmplx-pkgs-dbg_ipro.log]
>>  /Users/petsc/petsc.next-3/src/sys/utils/pdisplay.c:136:12: warning: 
>> unused variable 'len' [-Wunused-variable]
>>
>> [http://ftp.mcs.anl.gov/pub/petsc/nightlylogs/archive/2018/03/05/build_next_arch-linux-opt-cxx-quad_grind.log]
>>  /sandbox/petsc/petsc.next-3/src/sys/utils/pdisplay.c:136:12: warning: 
>> unused variable 'len' [-Wunused-variable]
>>
>> [http://ftp.mcs.anl.gov/pub/petsc/nightlylogs/archive/2018/03/05/build_next_arch-c-exodus-dbg-builder_es.log]
>>  /sandbox/petsc/petsc.next-3/src/sys/utils/pdisplay.c:136:12: warning: 
>> unused variable 'len' [-Wunused-variable]
>>
>> [http://ftp.mcs.anl.gov/pub/petsc/nightlylogs/archive/2018/03/05/build_next_arch-linux-xsdk-dbg_cg.log]
>>  /sandbox/petsc/petsc.next/src/sys/utils/pdisplay.c:136:12: warning: 
>> unused variable 'len' [-Wunused-variable]
>>
>> [http://ftp.mcs.anl.gov/pub/petsc/nightlylogs/archive/2018/03/05/build_next_arch-cuda-double_es.log]
>>  /sandbox/petsc/petsc.next/src/sys/utils/pdisplay.c:136:12: warning: 
>> unused variable 'len' [-Wunused-variable]
>>
>> [http://ftp.mcs.anl.gov/pub/petsc/nightlylogs/archive/2018/03/05/build_next_arch-linux-matlab-ilp64-gcov_thrash.log]
>>  /sandbox/petsc/petsc.next/src/sys/utils/pdisplay.c:136:12: warning: 
>> unused variable 'len' [-Wunused-variable]
>>
>> [http://ftp.mcs.anl.gov/pub/petsc/nightlylogs/archive/2018/03/05/build_next_arch-linux-pkgs-opt_crank.log]
>>  /sandbox/petsc/petsc.next/src/sys/utils/pdisplay.c:136:12: warning: 
>> unused variable 'len' [-Wunused-variable]
>>
>> [http://ftp.mcs.anl.gov/pub/petsc/nightlylogs/archive/2018/03/05/build_next_arch-viennacl_es.log]
>>  /sandbox/petsc/petsc.next-4/src/sys/utils/pdisplay.c:136:12: warning: 
>> unused variable 'len' [-Wunused-variable]
>>
>> [http://ftp.mcs.anl.gov/pub/petsc/nightlylogs/archive/2018/03/05/build_next_arch-osx-xsdk-opt_ipro.log]
>>  /Users/petsc/petsc.next-4/src/sys/utils/pdisplay.c:136:12: warning: 
>> unused variable 'len' [-Wunused-variable]
>>
>> [http://ftp.mcs.anl.gov/pub/petsc/nightlylogs/archive/2018/03/05/build_next_arch-linux-pkgs-dbg-ftn-interfac

Re: [petsc-dev] Rename csrperm files to aijperm?

[Smith, Barry F.]

Fine

> On Mar 5, 2018, at 8:18 PM, Richard Tran Mills  wrote:
> 
> Is there any good reason behind keeping the 'csrperm' in file and directory 
> names when MATCSRPERM was changed to MATAIJPERM? If not, I'm going to rename 
> things to 'aijperm' to stay consistent with the type name.
> 
> --Richard

Re: [petsc-dev] change in Manual Pages generation?

[Smith, Barry F.]

Yes it was part of a set of changes to the manual pages. I can understand 
why you prefer the original format.



 Barry


> On Mar 8, 2018, at 5:06 AM, Vaclav Hapla  wrote:
> 
> I have noticed that Level and Location is generated in a different way in 
> maint and master, compare e.g.
> http://www.mcs.anl.gov/petsc/petsc-dev/docs/manualpages/IS/PetscSectionCopy.html
> http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/IS/PetscSectionCopy.html
> 
> Is it a deliberate change?
> I find the original way better.
> 
> Vaclav

Re: [petsc-dev] PETSc blame digest (next) 2018-03-08

[Smith, Barry F.]

  Fixed


> On Mar 8, 2018, at 9:00 AM, PETSc checkBuilds  
> wrote:
> 
> 
> 
> Dear PETSc developer,
> 
> This email contains listings of contributions attributed to you by
> `git blame` that caused compiler errors or warnings in PETSc automated
> testing.  Follow the links to see the full log files. Please attempt to fix
> the issues promptly or let us know at petsc-dev@mcs.anl.gov if you are unable
> to resolve the issues.
> 
> Thanks,
>  The PETSc development team
> 
> 
> 
> warnings attributed to commit 
> https://bitbucket.org/petsc/petsc/commits/e885f1a
> Move logic from -snes_type test to be called instead with each computation of 
> the Jacobian
> 
>  src/snes/interface/snes.c:2284
>
> [http://ftp.mcs.anl.gov/pub/petsc/nightlylogs/archive/2018/03/08/build_next_arch-linux-c89_el6.log]
>  /home/sandbox/petsc/petsc.next-2/src/snes/interface/snes.c:2284:3: 
> warning: ISO C90 forbids mixed declarations and code [-Wpedantic]
> 
> 
> To opt-out from receiving these messages - send a request to 
> petsc-dev@mcs.anl.gov.

Re: [petsc-dev] [petsc-checkbuilds] PETSc blame digest (next) 2018-03-10

[Smith, Barry F.]

  Siegried,

 You need to do a build with --with-clanguage=cxx 
--with-scalar-type=complex and fix all the errors that come.

   Barry


> On Mar 10, 2018, at 9:00 AM, PETSc checkBuilds 
>  wrote:
> 
> 
> 
> Dear PETSc developer,
> 
> This email contains listings of contributions attributed to you by
> `git blame` that caused compiler errors or warnings in PETSc automated
> testing.  Follow the links to see the full log files. Please attempt to fix
> the issues promptly or let us know at petsc-dev@mcs.anl.gov if you are unable
> to resolve the issues.
> 
> Thanks,
>  The PETSc development team
> 
> 
> 
> warnings attributed to commit 
> https://bitbucket.org/petsc/petsc/commits/3a02854
> Add p(l)-CG KSP solver: pipelined Conjugate Gradient with deep pipelines
> 
>  src/ksp/ksp/impls/cg/pipelcg/pipelcg.c:233
>
> [http://ftp.mcs.anl.gov/pub/petsc/nightlylogs/archive/2018/03/10/build_next_arch-osx-10.6-cxx-cmplx-pkgs-dbg_ipro.log]
>  /Users/petsc/petsc.next-3/src/ksp/ksp/impls/cg/pipelcg/pipelcg.c:233:40: 
> error: invalid operands to binary expression ('complex' and 'int')
>
> [http://ftp.mcs.anl.gov/pub/petsc/nightlylogs/archive/2018/03/10/build_next_arch-linux-cmplx-single_steamroller.log]
>  
> /sandbox/petsc/petsc.next-3/src/ksp/ksp/impls/cg/pipelcg/pipelcg.c:233:40: 
> error: invalid operands to binary < (have 'PetscScalar' and 'int')
>
> [http://ftp.mcs.anl.gov/pub/petsc/nightlylogs/archive/2018/03/10/build_next_arch-linux-cxx-cmplx-pkgs-64idx_churn.log]
>  
> /sandbox/petsc/petsc.next-2/src/ksp/ksp/impls/cg/pipelcg/pipelcg.c:233:40: 
> error: invalid operands to binary expression ('complex' and 'int')
>
> [http://ftp.mcs.anl.gov/pub/petsc/nightlylogs/archive/2018/03/10/build_next_arch-linux-gcc-ifc-cmplx_crank.log]
>  
> /sandbox/petsc/petsc.next-2/src/ksp/ksp/impls/cg/pipelcg/pipelcg.c:233:40: 
> error: invalid operands to binary < (have 'PetscScalar' and 'int')
>
> [http://ftp.mcs.anl.gov/pub/petsc/nightlylogs/archive/2018/03/10/build_next_arch-linux-cmplx-gcov_steamroller.log]
>  
> /sandbox/petsc/petsc.next-2/src/ksp/ksp/impls/cg/pipelcg/pipelcg.c:233:40: 
> error: no match for 'operator<' (operand types are 'std::complex' and 
> 'int')
>
> [http://ftp.mcs.anl.gov/pub/petsc/nightlylogs/archive/2018/03/10/build_next_arch-freebsd-cxx-cmplx-pkgs-dbg_wii.log]
>  
> /usr/home/balay/petsc.next/src/ksp/ksp/impls/cg/pipelcg/pipelcg.c:233:40: 
> error: no match for 'operator<' (operand types are 'std::complex' and 
> 'int')
> 
>  src/ksp/ksp/impls/cg/pipelcg/pipelcg.c:343
>
> [http://ftp.mcs.anl.gov/pub/petsc/nightlylogs/archive/2018/03/10/build_next_arch-osx-10.6-cxx-cmplx-pkgs-dbg_ipro.log]
>  /Users/petsc/petsc.next-3/src/ksp/ksp/impls/cg/pipelcg/pipelcg.c:343:22: 
> error: assigning to 'PetscReal' (aka 'double') from incompatible type 
> 'PetscScalar' (aka 'complex')
>
> [http://ftp.mcs.anl.gov/pub/petsc/nightlylogs/archive/2018/03/10/build_next_arch-linux-cmplx-gcov_steamroller.log]
>  
> /sandbox/petsc/petsc.next-2/src/ksp/ksp/impls/cg/pipelcg/pipelcg.c:343:20: 
> error: cannot convert 'PetscScalar {aka std::complex}' to 'PetscReal 
> {aka double}' in assignment
>
> [http://ftp.mcs.anl.gov/pub/petsc/nightlylogs/archive/2018/03/10/build_next_arch-freebsd-cxx-cmplx-pkgs-dbg_wii.log]
>  
> /usr/home/balay/petsc.next/src/ksp/ksp/impls/cg/pipelcg/pipelcg.c:343:20: 
> error: cannot convert 'PetscScalar {aka std::complex}' to 'PetscReal 
> {aka double}' in assignment
>
> [http://ftp.mcs.anl.gov/pub/petsc/nightlylogs/archive/2018/03/10/build_next_arch-linux-cxx-cmplx-pkgs-64idx_churn.log]
>  
> /sandbox/petsc/petsc.next-2/src/ksp/ksp/impls/cg/pipelcg/pipelcg.c:343:22: 
> error: assigning to 'PetscReal' (aka 'double') from incompatible type 
> 'PetscScalar' (aka 'complex')
> 
>  src/ksp/ksp/impls/cg/pipelcg/pipelcg.c:358
>
> [http://ftp.mcs.anl.gov/pub/petsc/nightlylogs/archive/2018/03/10/build_next_arch-osx-10.6-cxx-cmplx-pkgs-dbg_ipro.log]
>  /Users/petsc/petsc.next-3/src/ksp/ksp/impls/cg/pipelcg/pipelcg.c:358:22: 
> error: no matching function for call to 'fabs'
> 
> 
> To opt-out from receiving these messages - send a request to 
> petsc-dev@mcs.anl.gov.

[petsc-dev] dm/interface directory files should NOT have DMPlex code in them. Shudder!

[Smith, Barry F.]

Re: [petsc-dev] dm/interface directory files should NOT have DMPlex code in them. Shudder!

[Smith, Barry F.]

Please implement things properly with subclasses (subtypes) and get this crap 
out of the public interface files

dm/interfaces should be able to be compiled and linked even if dm/impls stuff 
did not exist. This mixing of particular subclass implementations and public 
interface functions is disgusting. Ideally we'd have nightly tests that flags 
this broken stuff.



  Barry

#include 

  {
PetscBool isplex;

ierr = PetscObjectTypeCompare((PetscObject) cdm, DMPLEX, 
&isplex);CHKERRQ(ierr);
if (isplex) {
  ierr = DMPlexGetHeightStratum(cdm, 0, &cStart, &cEnd);CHKERRQ(ierr);
} else SETERRQ(PetscObjectComm((PetscObject) cdm), PETSC_ERR_ARG_WRONG, 
"Coordinate localization requires a DMPLEX coordinate DM");

PetscBool isplex;

ierr = PetscObjectTypeCompare((PetscObject) cdm, DMPLEX, 
&isplex);CHKERRQ(ierr);
if (isplex) {
  ierr = DMPlexGetDepthStratum(cdm, 0, &vStart, &vEnd);CHKERRQ(ierr);
  ierr = DMPlexGetMaxProjectionHeight(cdm,&maxHeight);CHKERRQ(ierr);
  ierr = DMGetWorkArray(dm,2*(maxHeight + 
1),MPIU_INT,&pStart);CHKERRQ(ierr);
  pEnd = &pStart[maxHeight + 1];
  newStart = vStart;
  newEnd   = vEnd;
  for (h = 0; h <= maxHeight; h++) {
ierr = DMPlexGetHeightStratum(cdm, h, &pStart[h], 
&pEnd[h]);CHKERRQ(ierr);
newStart = PetscMin(newStart,pStart[h]);
newEnd   = PetscMax(newEnd,pEnd[h]);



> On Mar 11, 2018, at 4:21 PM, Matthew Knepley  wrote:
> 
> This stuff is in DM because two concepts need to be promoted to DM. They are
> already both in DA and Plex:
> 
> 1) GlobalToNatural, which is in dmi.c
> 
> 2) Coordinates, in particular periodic coordinates, which is in dm.c
> 
> Removing these means adding 1 or 2 functions to dmimpl
> 
>Matt
> 
> 
> On Mon, Mar 12, 2018 at 3:06 AM, Smith, Barry F.  wrote:
> 
> 
> 
> 
> -- 
> What most experimenters take for granted before they begin their experiments 
> is infinitely more interesting than any results to which their experiments 
> lead.
> -- Norbert Wiener
> 
> https://www.cse.buffalo.edu/~knepley/

Re: [petsc-dev] dm/interface directory files should NOT have DMPlex code in them. Shudder!

[Smith, Barry F.]

> On Mar 11, 2018, at 6:49 PM, Matthew Knepley  wrote:
> 
> On Mon, Mar 12, 2018 at 8:33 AM, Smith, Barry F.  wrote:
> Please implement things properly with subclasses (subtypes) and get this crap 
> out of the public interface files
> 
> See below.

   I don't understand See Below. why are there not functions in the function 
table for these operations so the functions can be implemented back in the 
impls directory where they belong. Why do DMPlex calls appear directly in this 
function which should not know about particular implementations.

> 
>Matt
>  
> dm/interfaces should be able to be compiled and linked even if dm/impls stuff 
> did not exist. This mixing of particular subclass implementations and public 
> interface functions is disgusting. Ideally we'd have nightly tests that flags 
> this broken stuff.
> 
> 
> 
>   Barry
> 
> #include 
> 
>   {
> PetscBool isplex;
> 
> ierr = PetscObjectTypeCompare((PetscObject) cdm, DMPLEX, 
> &isplex);CHKERRQ(ierr);
> if (isplex) {
>   ierr = DMPlexGetHeightStratum(cdm, 0, &cStart, &cEnd);CHKERRQ(ierr);
> } else SETERRQ(PetscObjectComm((PetscObject) cdm), PETSC_ERR_ARG_WRONG, 
> "Coordinate localization requires a DMPLEX coordinate DM");
> 
> PetscBool isplex;
> 
> ierr = PetscObjectTypeCompare((PetscObject) cdm, DMPLEX, 
> &isplex);CHKERRQ(ierr);
> if (isplex) {
>   ierr = DMPlexGetDepthStratum(cdm, 0, &vStart, &vEnd);CHKERRQ(ierr);
>   ierr = DMPlexGetMaxProjectionHeight(cdm,&maxHeight);CHKERRQ(ierr);
>   ierr = DMGetWorkArray(dm,2*(maxHeight + 
> 1),MPIU_INT,&pStart);CHKERRQ(ierr);
>   pEnd = &pStart[maxHeight + 1];
>   newStart = vStart;
>   newEnd   = vEnd;
>   for (h = 0; h <= maxHeight; h++) {
> ierr = DMPlexGetHeightStratum(cdm, h, &pStart[h], 
> &pEnd[h]);CHKERRQ(ierr);
> newStart = PetscMin(newStart,pStart[h]);
> newEnd   = PetscMax(newEnd,pEnd[h]);
> 
> 
> 
> > On Mar 11, 2018, at 4:21 PM, Matthew Knepley  wrote:
> >
> > This stuff is in DM because two concepts need to be promoted to DM. They are
> > already both in DA and Plex:
> >
> > 1) GlobalToNatural, which is in dmi.c
> >
> > 2) Coordinates, in particular periodic coordinates, which is in dm.c
> >
> > Removing these means adding 1 or 2 functions to dmimpl
> >
> >Matt
> >
> >
> > On Mon, Mar 12, 2018 at 3:06 AM, Smith, Barry F.  wrote:
> >
> >
> >
> >
> > --
> > What most experimenters take for granted before they begin their 
> > experiments is infinitely more interesting than any results to which their 
> > experiments lead.
> > -- Norbert Wiener
> >
> > https://www.cse.buffalo.edu/~knepley/
> 
> 
> 
> 
> -- 
> What most experimenters take for granted before they begin their experiments 
> is infinitely more interesting than any results to which their experiments 
> lead.
> -- Norbert Wiener
> 
> https://www.cse.buffalo.edu/~knepley/

Re: [petsc-dev] Test parsers is parsing *.c~ files

[Smith, Barry F.]

  This drives me nuts also, obviously Scott does not use Emacs or he would have 
make sure long ago that the test harness would not trip over the crumbs of 
Emacs.

   Matt, You'll need to share the exact forms to of the crumbs that are 
confusing the test harness.

   Barry

> On Mar 14, 2018, at 11:03 PM, Matthew Knepley  wrote:
> 
> This took me forever to find. The parser is parsing emacs backups and 
> overwriting some tests.
> 
>Matt
> 
> -- 
> What most experimenters take for granted before they begin their experiments 
> is infinitely more interesting than any results to which their experiments 
> lead.
> -- Norbert Wiener
> 
> https://www.cse.buffalo.edu/~knepley/

Re: [petsc-dev] Nested event logging and human friendly output

[Smith, Barry F.]

   It is in 3.8 please let us know if you have any trouble with it.

   Barry


> On Mar 15, 2018, at 3:03 AM, Klaij, Christiaan  wrote:
> 
> Hi Barry,
> 
> It's been a while and I'll soon be upgrading our production
> software from petsc-3.7.5 to petsc-3.8.3. Can you tell me if the
> nested event logging made it into 3.8? If so, I would probably
> remove our own version in favor of the petsc maintained
> version. 
> 
> Best regards,
> Chris
> 
> dr. ir. Christiaan Klaij | Senior Researcher | Research & Development
> MARIN | T +31 317 49 33 44 | c.kl...@marin.nl | www.marin.nl
> 
>
> MARIN news: Numerical study of cavitation on a NACA0015 hydrofoil: solution 
> verification
> From: Koos Huijssen 
> Sent: Friday, August 12, 2016 8:07 AM
> To: Barry Smith
> Cc: petsc-dev@mcs.anl.gov; Klaij, Christiaan; Bas van 't Hof
> Subject: Re: [petsc-dev] Nested event logging and human friendly output
>  
> Hi Barry,
> 
> Looking forward to that!
> 
> Koos
> From: Barry Smith 
> Sent: Friday, August 12, 2016 4:02:42 AM
> To: Koos Huijssen
> Cc: petsc-dev@mcs.anl.gov; Klaij, Christiaan; Bas van 't Hof
> Subject: Re: [petsc-dev] Nested event logging and human friendly output
>  
> 
>   Thanks. This will be in master soon.
> 
>Barry
> 
> > On Aug 8, 2016, at 5:15 AM, Koos Huijssen  wrote:
> > 
> > Hi Barry,
> > 
> > The fix is to replace the "j<=depth" on line 736 with "j > should read
> > 
> > for (j=0; same && j > nstPath[j]) ? PETSC_TRUE : PETSC_FALSE;}
> > 
> > That should resolve the valgrind issue.
> > 
> > With kind regards,
> > 
> > Koos
> > 
> > 
> > -Original Message-
> > From: Barry Smith [mailto:bsm...@mcs.anl.gov] 
> > Sent: vrijdag 5 augustus 2016 22:39
> > To: Koos Huijssen 
> > Cc: petsc-dev@mcs.anl.gov; Klaij, Christiaan ; Bas van 't 
> > Hof 
> > Subject: Re: [petsc-dev] Nested event logging and human friendly output
> > 
> > 
> >There appears to be an error indicated by valgrind at: 
> > ftp://ftp.mcs.anl.gov/pub/petsc/nightlylogs/archive/2016/08/04/examples_master_arch-linux-pkgs-valgrind_grind.log
> > 
> > Note that the nstPath[] arrays only seem to filled up to but not including 
> > depth but the  "same" loop has j<=depth. Could you please clarify how I 
> > should fix this?
> > 
> > 
> >  if (i >for (j=0; j > tree[i].nstPath[j];
> >for (j=tree[i].depth; j > illegalEvent;
> >  } else {
> >for (j=0; j > illegalEvent;
> >  }
> > 
> >  /* Communicate with other processes to obtain the next path and its 
> > depth */
> >  ierr = MPIU_Allreduce(nstMyPath, nstPath, depth, MPI_INT, MPI_MIN, 
> > comm);CHKERRQ(ierr);
> >  for (j=depth-1; (int) j>=0; j--) { 
> >if (nstPath[j]==illegalEvent) depth=j;
> >  }
> > 
> >  if (depth>0) {
> >/* If the path exists */
> > 
> >/* check whether the next path is the same as this process's next 
> > path */
> >same = PETSC_TRUE; 
> >for (j=0; same && j<=depth; j++) { same = (same &&  nstMyPath[j] == 
> > nstPath[j]) ? PETSC_TRUE : PETSC_FALSE;}
> > 
> > 
> > 
> >> On Aug 4, 2016, at 5:01 PM, Koos Huijssen  wrote:
> >> 
> >> Hi Barry,
> >> 
> >> I did some analysis on the results, but I see nothing strange. The missing 
> >> MatAssemblyBegin could be because its time falls under the threshold of 
> >> 0.01% of the total runtime, in which case it is left out of the tree. I 
> >> ran the same case, and I got both Begin/End operations, but both at 0% (so 
> >> probably both dwindling around the 0.01% threshold). The 
> >> MatAssemblyBegin/MatAssemblyEnd pair are part of the SNESSolve routine, 
> >> but they are not located within the SNES_Solve log event. This event only 
> >> registers the time spent in the solve routine in snes->ops->solve. If I 
> >> set an event around the call to SNESSolve in ex19.c, they do appear within 
> >> that event. Maybe something to do with MatInterpolate or DMInterpolate?
> >> 
> >> Since the log events have never been used in a nested logging before, it 
> >> could be that this type of misunderstanding will occur more often whenever 
> >> a log event is not one-to-one coupled to its routine but to a section 
> >> within that routine.
> >> 
> >> With kind regards,
> >> 
> >> Koos
> >> 
> >> From: Koos Huijssen
> >> Sent: donderdag 4 augustus 2016 22:12
> >> To: Koos Huijssen 
> >> Subject: Fwd: [petsc-dev] Nested event logging and human friendly 
> >> output
> >> 
> >> 
> >> 
> >> 
> >> Begin doorgestuurd bericht:
> >> 
> >> Van: Barry Smith 
> >> Datum: 17 juli 2016 04:44:04 CEST
> >> Aan: Koos Huijssen 
> >> Kopie: Richard Mills , 
> >> "petsc-dev@mcs.anl.gov" , "Klaij, Christiaan" 
> >> , "Bas van 't Hof" 
> >> Onderwerp: Antw.: [petsc-dev] Nested event logging and human friendly 
> >> output
> >> 
> >> 
> >>  Thank you very much for fixing the errors I introduced and improving the 
> >> code. I have put it into branch barry/fix-xml-logging and merged to next 
> >> for testing. 
> >> 
> >>  I do have

Re: [petsc-dev] Test parsers is parsing *.c~ files

[Smith, Barry F.]

> On Mar 15, 2018, at 1:22 PM, Scott Kruger  wrote:
> 
> 
> 
> Yes, as you surmise Barry, I am a vim user.
> 
> Based on earlier comments, I have this in config/gmakegentest.py:
> 
># Ignore emacs and other temporary files
>if exfile.startswith("."): continue
>if exfile.startswith("#"): continue
> 
> I assume that this:
>if exfile.endwith("~"): continue
> 
> will work.   Do folks see any problems?

   No, there may be others also.

> 
> Scott
> 
> 
> 
> On 3/14/18 10:05 PM, Smith, Barry F. wrote:
>>   This drives me nuts also, obviously Scott does not use Emacs or he would 
>> have make sure long ago that the test harness would not trip over the crumbs 
>> of Emacs.
>>Matt, You'll need to share the exact forms to of the crumbs that are 
>> confusing the test harness.
>>Barry
>>> On Mar 14, 2018, at 11:03 PM, Matthew Knepley  wrote:
>>> 
>>> This took me forever to find. The parser is parsing emacs backups and 
>>> overwriting some tests.
>>> 
>>>Matt
>>> 
>>> -- 
>>> What most experimenters take for granted before they begin their 
>>> experiments is infinitely more interesting than any results to which their 
>>> experiments lead.
>>> -- Norbert Wiener
>>> 
>>> https://www.cse.buffalo.edu/~knepley/
> 
> -- 
> Tech-X Corporation   kru...@txcorp.com
> 5621 Arapahoe Ave, Suite A   Phone: (720) 974-1841
> Boulder, CO 80303Fax:   (303) 448-7756

Re: [petsc-dev] Test Code Coverage (gcov) page is missing

[Smith, Barry F.]

  Any suggestions on how to achieve this?


> On Mar 16, 2018, at 11:15 AM, Junchao Zhang  wrote:
> 
> For each tested line, it would be helpful if we also show the test (one is 
> enough) that tested the line.
> 
> --Junchao Zhang
> 
> On Fri, Mar 16, 2018 at 10:40 AM, Balay, Satish  wrote:
> On Fri, 16 Mar 2018, Junchao Zhang wrote:
> 
> > The coverage page at http://ftp.mcs.anl.gov/pub/petsc/nightlylogs/index.html
> > is missing. When I read certain PETSc code, I feel some if conditions will
> > never be met (i.e., there are dead statements). To confirm that, I want to
> > know which tests run into that condition.
> 
> Thanks - its updated now.
> 
> 
> Jed,
> 
> I must have messedup the testing of the latest gcov changes..
> 
> >>>
> [thwomp:petsc/nightlylogs/gcov] petsc> cat 
> ~/nightlylogs/archive/2018/03/16/gcov-snapshot-update.log
> Traceback (most recent call last):
>   File "/sandbox/petsc/petsc.clone/lib/petsc/bin/maint/gcov.py", line 18, in 
> 
> PETSC_ARCH = os.environ['PETSC_ARCH']
>   File "/usr/lib/python2.7/UserDict.py", line 23, in __getitem__
> raise KeyError(key)
> KeyError: 'PETSC_ARCH'
> <<<
> 
> So the issue is - this script is run in 2 different modes:
> 
> lib/petsc/bin/maint/gcov.py:print " ./gcov.py -run_gcov  "
> lib/petsc/bin/maint/gcov.py:print " ./gcov.py -merge_gcov 
> [LOC] tarballs"
> 
> For the second mode is with [LOC] - i.e there is no prior build - i.e no 
> PETSC_ARCH
> 
> So how about the following change?
> 
> diff --git a/lib/petsc/bin/maint/gcov.py b/lib/petsc/bin/maint/gcov.py
> index 6d0e1e0..0a49978 100755
> --- a/lib/petsc/bin/maint/gcov.py
> +++ b/lib/petsc/bin/maint/gcov.py
> @@ -14,10 +14,6 @@ import operator
>  import sys
>  from   time import gmtime,strftime
> 
> -PETSC_DIR = os.environ['PETSC_DIR']
> -PETSC_ARCH = os.environ['PETSC_ARCH']
> -OBJDIR = os.path.join(PETSC_DIR, PETSC_ARCH, 'obj')
> -
>  def run_gcov(gcov_dir):
> 
>  # 1. Runs gcov
> @@ -389,11 +385,15 @@ def make_htmlpage(gcov_dir,LOC,tarballs):
> 
>  def main():
> 
> +global PETSC_DIR,PETSC_ARCH,OBJDIR
>  USER = os.environ['USER']
>  gcov_dir = "/tmp/gcov-"+USER
> +PETSC_DIR = os.environ['PETSC_DIR']
> 
>  if (sys.argv[1] == "-run_gcov"):
>  print "Running gcov and creating tarball"
> +PETSC_ARCH = os.environ['PETSC_ARCH']
> +OBJDIR = os.path.join(PETSC_DIR, PETSC_ARCH, 'obj')
>  run_gcov(gcov_dir)
>  make_tarball(gcov_dir)
>  elif (sys.argv[1] == "-merge_gcov"):
> 
> 
> Satish
> 

Re: [petsc-dev] Nested event logging and human friendly output

[Smith, Barry F.]

> On Mar 16, 2018, at 6:06 AM, Koos Huijssen  wrote:
> 
> A solution seems to be that the style sheet is present on a public server, so 
> that it can be retrieved using http. This should be fine for all browsers. 

   This does not work for Firefox (generates an error code) Safari (rejected as 
unsafe since does not come from the same domain as the input file) or Chrome 
(also rejected as unsafe).

   Putting the style file in the same directory as the xml file works for 
Firefox but is rejected as unsafe by Chrome and Safari (does not look possible 
to pass obscure command line arguments to Chrome on the Mac).

So it appears next to hopeless (apparently the browsers really want you to 
be retrieving everything via a web sever to view them).

I found that that one can embed directly into a file some style information 
with the  . I tried this with your style file and it didn't work 
(I suspect because your style file uses all kinds of cool stuff not supported 
by  .)  If you can figure out how to include all the style 
information directly into the generated file that would be the ideal solution.



   Barry


 


> Maybe somewhere on https://www.mcs.anl.gov/petsc ...?
> 
> Regards,
> 
> Koos
> 
> On 03/16/2018 12:04 PM, Klaij, Christiaan wrote:
>> Thanks Koos! We have the same problem on microsoft internet explorer. Safari 
>> does some magic for sure.
>> 
>> Chris
>> 
>> dr. ir.Christiaan Klaij | Senior Researcher | Research & Development
>> MARIN | T +31 317 49 33 44 | c.kl...@marin.nl | www.marin.nl
>> 
>>
>> MARIN news: Putting a new spin on propeller design
>> From: Koos Huijssen 
>> Sent: Friday, March 16, 2018 12:02 PM
>> To: Klaij, Christiaan; Smith, Barry F.
>> Cc: petsc-dev@mcs.anl.gov
>> Subject: Re: [petsc-dev] Nested event logging and human friendly output
>>  
>> Hi Chris,
>> Indeed, this is exactly the issue. When we investigated this it seemed to 
>> work on Barry's system, but it has never worked on mine either. The browsers 
>> do not allow hard paths to be used for the xml style sheet.
>> For Chrome there is an extra thing. If you launch Chrome with the flag 
>> '--allow-file-access-from-files' it should give the desired result (see 
>> here).
>> A note in the manual would surely be a good thing.
>> With kind regards,
>> Koos
>> 
>> On 03/16/2018 11:54 AM, Klaij, Christiaan wrote:
>>> Barry,
>>> 
>>> I've tried it quickly with petsc-3.8.3 and
>>> snes/examples/tutorials/ex70.c and it seems to work except for
>>> the stylesheet. The xml log has this header in my case:
>>> 
>>> >> href="/home/cklaij/ReFRESCO/Dev/trunk/Libs/build/petsc-3.8.3-dbg/share/petsc/xml/performance_xml2html.xsl"?>
>>> 
>>> Even though the path is correct, both Firefox and Chrome on linux
>>> do not load the stylesheet. I have to
>>> 
>>> 1) copy "performance_xml2html.xsl" to the directory which
>>> contains the xml log (a symlink won't do),
>>> 
>>> 2) then edit the header to href="performance_xml2html.xsl".
>>> 
>>> These steps work for Firefox, but not for Chrome (it just shows a
>>> blank page).
>>> 
>>> If I remember correctly (Koos?) then this problem was due to some
>>> browser safety stuff which can't be avoided. If this is still
>>> true and not fixable, we should probably add a note in the manual
>>> explaining both steps?
>>> 
>>> Chris
>>> 
>>> 
>>> 
>>> dr. ir. Christiaan Klaij  | Senior Researcher | Research & Development
>>> MARIN | T +31 317 49 33 44 | 
>>> mailto:c.kl...@marin.nl | 
>>> https://emea01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.marin.nl&data=02%7C01%7Ckoos.huijssen%40vortech.nl%7Cf467bc9cbee84636730208d58b2c45c1%7C5fe8f8070fe44cdf8dc636475a0b8555%7C0%7C0%7C636567944636671153&sdata=wf%2BYdEvdpvMh9aK7D51yBDHFQvlziBS9U3ArOA85etE%3D&reserved=0
>>> 
>>> 
>>> MARIN news: 
>>> https://emea01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.marin.nl%2Fweb%2FNews%2FNews-items%2FFrequency-of-spill-model-for-area-risk-assessment-of-shipsource-oil-spills-in-Canadian-waters-1.htm&data=02%7C01%7Ckoos.huijssen%40vortech.nl%7Cf467bc9cbee84636730208d58b2c45c1%7C5fe8f8070fe44cdf8dc636475a0b8555%7C0%7C0%7C636567944636671153&sdata=2TVwn1x5EZuNHzVHCMO1BLNNRMv%2BrSe4eOJ3tQ7vhJg%3D&reserved=0
>>> 
>>> 
>>> 
>>> From: Smith, Barry F. 
>>> 
>>> 
>>&g

Re: [petsc-dev] Is ./configure --help broken?

[Smith, Barry F.]

  I think RDict.db is valuable, but saving it every five seconds while 
configure is running is not valuable and should be changed to just save it at 
the end of the run.

   Barry


> On Mar 16, 2018, at 8:14 PM, Matthew Knepley  wrote:
> 
> On Fri, Mar 16, 2018 at 6:25 PM, Jed Brown  wrote:
> Matthew Knepley  writes:
> 
> > On Fri, Mar 16, 2018 at 2:20 PM, Jed Brown  wrote:
> >
> >> Matthew Knepley  writes:
> >>
> >> > On Fri, Mar 16, 2018 at 1:27 PM, Jed Brown  wrote:
> >> >
> >> >> Matthew Knepley  writes:
> >> >>
> >> >> > On Fri, Mar 16, 2018 at 1:17 PM, Jed Brown  wrote:
> >> >> >
> >> >> >> Matthew Knepley  writes:
> >> >> >>
> >> >> >> > I agree. We should remove all code (about 2/3 of it) which does a
> >> >> >> > hierarchy of communicating dicts (the original design). That would
> >> >> >> > make everything simple.  No threads, no parents, etc. We leave in
> >> the
> >> >> >> > help the way we want it, types for args, etc. One thing its notably
> >> >> >> > missing, and that PETSc Options are missing, is listing the thing
> >> that
> >> >> >> > set the option (default, command line, code, env).
> >> >> >>
> >> >> >> Does RDict even need to be persistent?  Who all reads it?  I wonder
> >> if
> >> >> >> an existing human-readable file would be sufficient instead?
> >> >> >>
> >> >> >
> >> >> > I think we should persist the entire set of options used to configure
> >> for
> >> >> > later
> >> >> > interrogation, however we have not done that much so far.
> >> >>
> >> >> CONFIGURE_OPTIONS is written to petscvariables and printed by make info.
> >> >> I think fewer duplications is desirable.
> >> >>
> >> >
> >> > This gets into a separate discussion. I think Python info is more useful
> >> > since its
> >> > directly visible to scripts we might write.
> >>
> >> Just call your Python parsing function.
> >
> >
> > Parse a makefile to get info? This is too convoluted for my first choice.
> >
> >
> >> But this gets back to my earlier question: who needs to read RDict.db and
> >> for what purpose?
> >>
> >
> > 1) We read this to get the configure modules during install and other post
> > configure operations.
> >
> > 2) I would like us to read RDict.db to get the original configure
> > environment
> 
> The problem is that RDict.db is opaque
> 
> This seems to be a normative characterization. I think the make variables are
> "dead" because its hard to make use of them in scripting.
>  
> and some people edit
> petscvariables and/or petscconf.h to work around weird issues.
> 
> Awful. We should not encourage this. It should flow from the top.
>  
>   We can't
> practically get rid of petscvariables without abandoning make.
> 
> The aim should not be to get rid of them, but rather to have them be read 
> only.
>  
> It's also frustrating to debug because the names used in Python may not
> match the names used in human-readable output,
> 
> Changing this is trivial.
> 
>   Matt
>  
> and that also prevents
> grepping to find unused variables.
> 
> 
> 
> -- 
> What most experimenters take for granted before they begin their experiments 
> is infinitely more interesting than any results to which their experiments 
> lead.
> -- Norbert Wiener
> 
> https://www.cse.buffalo.edu/~knepley/

Re: [petsc-dev] Tests are not working for me in next

[Smith, Barry F.]

> On Mar 19, 2018, at 2:51 PM, Jed Brown  wrote:
> 
> Satish Balay  writes:
> 
>> On Mon, 19 Mar 2018, Jed Brown wrote:
>> 
>>> Satish Balay  writes:
>>> 
 On Mon, 19 Mar 2018, Satish Balay wrote:
 
> On Mon, 19 Mar 2018, Jed Brown wrote:
> 
>> Satish Balay  writes:
>> 
 Thanks for the diagnosis.  Can we use @rpath/libpetsc.3.08.dylib?  Then
 we wouldn't need to run install_name_tool either.  If we want the
 absolute path written into the dylib, I can do that.
>>> 
>>> You mean '-install_name ${PREFIXDIR}/lib/libpetsc.3.08.dylib'? Yes - 
>>> this should eliminate the need for install_name_tool in 'make install' 
>>> [and that code can be removed]
>> 
>> No, I mean -install_name @rpath/libpetsc.3.08.dylib.
> 
> 
> mpicc -Wl,-multiply_defined,suppress -Wl,-multiply_defined -Wl,suppress 
> -Wl,-commons,use_dylibs -Wl,-search_paths_first -Wl,-no_compact_unwind
> -Wall -Wwrite-strings -Wno-strict-aliasing -Wno-unknown-pragmas 
> -fstack-protector -Qunused-arguments -fvisibility=hidden -g3  -dynamiclib 
> -install_name -install_name @rpath/libpetsc.3.08.dylib 
> -compatibility_version 3.08 -current_version 3.08.3 -single_module 
> -multiply_defined suppress -undefined dynamic_lookup -o 
> arch-prefix/lib/libpetsc.3.08.3.dylib ...
> 
> clang: error: no such file or directory: '@rpath/libpetsc.3.08.dylib'
> make[2]: *** [arch-prefix/lib/libpetsc.3.08.3.dylib] Error 1
 
 Ops - I have -install_name option listed twice.
 
 removing the dupliate..
 
 balay@ipro^~/petsc(next) $ make PETSC_DIR=/Users/balay/pinstall 
 PETSC_ARCH="" test
 Running test examples to verify correct installation
 Using PETSC_DIR=/Users/balay/pinstall and PETSC_ARCH=
 C/C++ example src/snes/examples/tutorials/ex19 run successfully with 1 MPI 
 process
 C/C++ example src/snes/examples/tutorials/ex19 run successfully with 2 MPI 
 processes
 Fortran example src/snes/examples/tutorials/ex5f run successfully with 1 
 MPI process
 Completed test examples
 =
 Now to evaluate the computer systems you plan use - do:
 make PETSC_DIR=/Users/balay/pinstall PETSC_ARCH= streams
 balay@ipro^~/petsc(next) $ 
 balay@ipro^~/petsc(next) $ otool -L ~/pinstall/lib/libpetsc.dylib 
 /Users/balay/pinstall/lib/libpetsc.dylib:
@rpath/libpetsc.3.08.dylib (compatibility version 3.8.0, current 
 version 3.8.3)

 /System/Library/Frameworks/Accelerate.framework/Versions/A/Frameworks/vecLib.framework/Versions/A/libLAPACK.dylib
  (compatibility version 1.0.0, current version 1.0.0)

 /System/Library/Frameworks/Accelerate.framework/Versions/A/Frameworks/vecLib.framework/Versions/A/libBLAS.dylib
  (compatibility version 1.0.0, current version 1.0.0)
/usr/lib/libc++.1.dylib (compatibility version 1.0.0, current version 
 400.9.0)
/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current 
 version 1252.0.0)
/Users/balay/soft/mpich-3.2/lib/libmpifort.12.dylib (compatibility 
 version 14.0.0, current version 14.0.0)
/Users/balay/soft/mpich-3.2/lib/libmpi.12.dylib (compatibility version 
 14.0.0, current version 14.0.0)
/Users/balay/soft/mpich-3.2/lib/libpmpi.12.dylib (compatibility version 
 14.0.0, current version 14.0.0)
/usr/local/opt/gcc/lib/gcc/7/libgfortran.4.dylib (compatibility version 
 5.0.0, current version 5.0.0)
/usr/local/lib/gcc/7/libgcc_s.1.dylib (compatibility version 1.0.0, 
 current version 1.0.0)
/usr/local/opt/gcc/lib/gcc/7/libquadmath.0.dylib (compatibility version 
 1.0.0, current version 1.0.0)
/Users/balay/soft/mpich-3.2/lib/libmpicxx.12.dylib (compatibility 
 version 14.0.0, current version 14.0.0)
 
 
 Ok - this works..
>>> 
>>> The consequence here is that users would be expected to use -Wl,-rpath,
>>> when linking to libpetsc.dylib, where now they probably get away without
>>> (on OSX).
>> 
>> Using '-install_name ${PREFIXDIR}/lib/libpetsc.3.08.dylib' would keep the 
>> current behavior.
> 
> Right, but that would prevent it from working in the build directory.

  I could care less about it working from the build directory. We don't test 
there anyways.

   Barry

> If you want both, you either need to relink or run install_name_tool as
> we currently do.  With @rpath, you don't need any of that.  There are
> arguments in favor of either convention.

Re: [petsc-dev] Nested event logging and human friendly output

[Smith, Barry F.]

  How does 
https://bitbucket.org/petsc/petsc/pull-requests/new?source=barry/docs-for-nested-xml-log-view&t=1
  look?

   Thanks

   Barry


> On Mar 20, 2018, at 4:40 AM, Koos Huijssen  wrote:
> 
> Dear Barry, Chris,
> 
> In that case in my opinion the best strategy would be to 
> 
> 1) Not have a full path in the stylesheet line of the xml output, so just use 
> the line
>   
> 2) Have a note in the manual that 
>   - Instructs the user to copy the style sheet next to the xml output 
> file.
>   - Explains that with this setup the user should be able to view the 
> reports using Firefox and Internet Explorer. For Chrome the option 
> --allow-file-access-from-files should be set. For Safari, local file access 
> should also be enabled (see 
> http://ccm.net/faq/36342-safari-how-to-enable-local-file-access)
>   - As a fall-back the user can also transform the xml to html using the 
> linux tool 'xsltproc' (see http://xmlsoft.org/XSLT/xsltproc2.html)
> 
> I have tested Firefox, IE, Chrome and Safari on our systems using the above 
> procedure (Windows: IE, Firefox and Chrome, Linux: Firefox, IOS: Safari) an 
> it works well. Previously I also tested xsltproc, and this also works.
> 
> If you wouldn't like this approach then we could also experiment with 
> embedding the stylesheet into the xml, see 
> http://www.dpawson.co.uk/xsl/sect2/onefile.html. But I currently do not have 
> the resources to do that.
> 
> With kind regards,
> 
> Koos 
> 
> -Original Message-
> From: Smith, Barry F. [mailto:bsm...@mcs.anl.gov] 
> Sent: vrijdag 16 maart 2018 19:55
> To: Koos Huijssen 
> Cc: Klaij, Christiaan ; petsc-dev@mcs.anl.gov
> Subject: Re: [petsc-dev] Nested event logging and human friendly output
> 
> 
> 
>> On Mar 16, 2018, at 6:06 AM, Koos Huijssen  wrote:
>> 
>> A solution seems to be that the style sheet is present on a public server, 
>> so that it can be retrieved using http. This should be fine for all 
>> browsers. 
> 
>   This does not work for Firefox (generates an error code) Safari (rejected 
> as unsafe since does not come from the same domain as the input file) or 
> Chrome (also rejected as unsafe).
> 
>   Putting the style file in the same directory as the xml file works for 
> Firefox but is rejected as unsafe by Chrome and Safari (does not look 
> possible to pass obscure command line arguments to Chrome on the Mac).
> 
>So it appears next to hopeless (apparently the browsers really want you to 
> be retrieving everything via a web sever to view them).
> 
>I found that that one can embed directly into a file some style 
> information with the  . I tried this with your style file and 
> it didn't work (I suspect because your style file uses all kinds of cool 
> stuff not supported by  .)  If you can figure out how to 
> include all the style information directly into the generated file that would 
> be the ideal solution.
> 
> 
> 
>   Barry
> 
> 
> 
> 
> 
>> Maybe somewhere on 
>> https://emea01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.mcs.anl.gov%2Fpetsc&data=02%7C01%7Ckoos.huijssen%40vortech.nl%7C66e17fb5b9c543b8bd2108d58b6f727e%7C5fe8f8070fe44cdf8dc636475a0b8555%7C0%7C0%7C636568233169349605&sdata=X%2FjZOjvsOlWR2NDbYjnQnBxocE2fe2zjSwcpgu%2F6Tp0%3D&reserved=0
>>  ...?
>> 
>> Regards,
>> 
>> Koos
>> 
>> On 03/16/2018 12:04 PM, Klaij, Christiaan wrote:
>>> Thanks Koos! We have the same problem on microsoft internet explorer. 
>>> Safari does some magic for sure.
>>> 
>>> Chris
>>> 
>>> dr. ir.Christiaan Klaij | Senior Researcher | Research & Development 
>>> MARIN | T +31 317 49 33 44 | c.kl...@marin.nl | 
>>> https://emea01.safelinks.protection.outlook.com/?url=www.marin.nl&dat
>>> a=02%7C01%7Ckoos.huijssen%40vortech.nl%7C66e17fb5b9c543b8bd2108d58b6f
>>> 727e%7C5fe8f8070fe44cdf8dc636475a0b8555%7C0%7C0%7C636568233169349605&
>>> sdata=5%2Bf0QNYiAYF4vYD8nwWXGS7aLPU1GkXoQvUPXutoB%2B4%3D&reserved=0
>>> 
>>>
>>>  MARIN news: Putting a new spin on propeller design
>>> From: Koos Huijssen 
>>> Sent: Friday, March 16, 2018 12:02 PM
>>> To: Klaij, Christiaan; Smith, Barry F.
>>> Cc: petsc-dev@mcs.anl.gov
>>> Subject: Re: [petsc-dev] Nested event logging and human friendly 
>>> output
>>> 
>>> Hi Chris,
>>> Indeed, this is exactly the issue. When we investigated this it seemed to 
>>> work on Barry's system, but it has never worked on mine either. The 
>>> browsers do not allow hard paths to

Re: [petsc-dev] Nested event logging and human friendly output

[Smith, Barry F.]

  Try this https://bitbucket.org/ 
petsc/petsc/commits/607d249ec66f5be42ddd7f6f35ea64c82fd126db   remove the 
spaces from the URL





> On Mar 21, 2018, at 3:58 AM, Koos Huijssen  wrote:
> 
> Hi Barry,
> 
> I couldn't gain access to the pull request. I have reconfigured by bitbucket 
> account, could you try this again?
> 
> Regards
> 
> 
> From: Smith, Barry F. 
> Sent: Tuesday, March 20, 2018 16:31
> To: Koos Huijssen
> Cc: Klaij, Christiaan; petsc-dev@mcs.anl.gov
> Subject: Re: [petsc-dev] Nested event logging and human friendly output
>  
> 
>   How does 
> https://emea01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fbitbucket.org%2Fpetsc%2Fpetsc%2Fpull-requests%2Fnew%3Fsource%3Dbarry%2Fdocs-for-nested-xml-log-view%26t%3D1&data=02%7C01%7Ckoos.huijssen%40vortech.nl%7C89ab8f1c6cf3429820c508d58e77af22%7C5fe8f8070fe44cdf8dc636475a0b8555%7C0%7C1%7C636571567074241616&sdata=YM86JOK2As2wRQJhTurtpm4z7u8VbqMkQ9L49It57cE%3D&reserved=0
>   look?
> 
>Thanks
> 
>Barry
> 
> 
> > On Mar 20, 2018, at 4:40 AM, Koos Huijssen  wrote:
> > 
> > Dear Barry, Chris,
> > 
> > In that case in my opinion the best strategy would be to 
> > 
> > 1) Not have a full path in the stylesheet line of the xml output, so just 
> > use the line
> >
> > 2) Have a note in the manual that 
> >- Instructs the user to copy the style sheet next to the xml output 
> > file.
> >- Explains that with this setup the user should be able to view the 
> > reports using Firefox and Internet Explorer. For Chrome the option 
> > --allow-file-access-from-files should be set. For Safari, local file access 
> > should also be enabled (see 
> > https://emea01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fccm.net%2Ffaq%2F36342-safari-how-to-enable-local-file-access&data=02%7C01%7Ckoos.huijssen%40vortech.nl%7C89ab8f1c6cf3429820c508d58e77af22%7C5fe8f8070fe44cdf8dc636475a0b8555%7C0%7C1%7C636571567074241616&sdata=wduvj5zczvFX%2FnxDH0kD%2B4Idl9M2Kmnv9jK2WOL84Kk%3D&reserved=0)
> >- As a fall-back the user can also transform the xml to html using 
> > the linux tool 'xsltproc' (see 
> > https://emea01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fxmlsoft.org%2FXSLT%2Fxsltproc2.html&data=02%7C01%7Ckoos.huijssen%40vortech.nl%7C89ab8f1c6cf3429820c508d58e77af22%7C5fe8f8070fe44cdf8dc636475a0b8555%7C0%7C1%7C636571567074241616&sdata=uud8nC7aTR9t8A47uALmb%2FkQC7Mgsf3HUUwM2hJKyzM%3D&reserved=0)
> > 
> > I have tested Firefox, IE, Chrome and Safari on our systems using the above 
> > procedure (Windows: IE, Firefox and Chrome, Linux: Firefox, IOS: Safari) an 
> > it works well. Previously I also tested xsltproc, and this also works.
> > 
> > If you wouldn't like this approach then we could also experiment with 
> > embedding the stylesheet into the xml, see 
> > https://emea01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.dpawson.co.uk%2Fxsl%2Fsect2%2Fonefile.html&data=02%7C01%7Ckoos.huijssen%40vortech.nl%7C89ab8f1c6cf3429820c508d58e77af22%7C5fe8f8070fe44cdf8dc636475a0b8555%7C0%7C1%7C636571567074241616&sdata=ZwKnULhzOVkgVve%2BgFihmwWObERIcL%2Flhgx%2FsXMzIwY%3D&reserved=0.
> >  But I currently do not have the resources to do that.
> > 
> > With kind regards,
> > 
> > Koos 
> > 
> > -Original Message-
> > From: Smith, Barry F. [mailto:bsm...@mcs.anl.gov] 
> > Sent: vrijdag 16 maart 2018 19:55
> > To: Koos Huijssen 
> > Cc: Klaij, Christiaan ; petsc-dev@mcs.anl.gov
> > Subject: Re: [petsc-dev] Nested event logging and human friendly output
> > 
> > 
> > 
> >> On Mar 16, 2018, at 6:06 AM, Koos Huijssen  
> >> wrote:
> >> 
> >> A solution seems to be that the style sheet is present on a public server, 
> >> so that it can be retrieved using http. This should be fine for all 
> >> browsers. 
> > 
> >   This does not work for Firefox (generates an error code) Safari (rejected 
> > as unsafe since does not come from the same domain as the input file) or 
> > Chrome (also rejected as unsafe).
> > 
> >   Putting the style file in the same directory as the xml file works for 
> > Firefox but is rejected as unsafe by Chrome and Safari (does not look 
> > possible to pass obscure command line arguments to Chrome on the Mac).
> > 
> >So it appears next to hopeless (apparently the browsers really want you 
> > to be retrieving everything via a web sever to view them).
> > 
> >I found that that one can embed directly into a file some style 
> > information with the

Re: [petsc-dev] --prefix install totally broken in next on Mac

[Smith, Barry F.]

> On Mar 21, 2018, at 7:41 PM, Jed Brown  wrote:
> 
> Satish Balay  writes:
> 
>> On Thu, 22 Mar 2018, Satish Balay wrote:
>> 
 make gnumake VERBOSE=1 does not show how the library is built, how does 
 one see what exactly is being passed to build the library? Impossible for 
 me to debug.
>> 
>> BTW: This is:
>> 
>> make V=1
> 
> Barry, see other thread about whether to use
> 
>  -install_name @rpath/libpetsc.so.3.08
> 
> versus the absolute path.  Using @rpath makes the library relocatable
> without changing it (like on Linux), but requires either linking with
> RPATH (-Wl,-rpath,/prefix/lib) or DYLD_LIBRARY_PATH (like on Linux).
> What is your preference?

I prefer the absolute path.

> 
 This was all fine in the past for prefix builds on the Mac all the recent 
 messing around broke it.

   Is there a branch that fixes the problem or has the problem just been 
commented on for a few days?

>> 
>> BTW: The changes didn't affect saws. It has its own installer that needs 
>> fixing?
> 
> Yeah, pretty sure Saws issues are unrelated.

   I'm not convinced.

Re: [petsc-dev] Manual builds fail on master

[Smith, Barry F.]

   I'll talk to Satish, it is now unclear (at least to me) where the manual is 
actually built and the output stored.


> On Mar 22, 2018, at 5:17 AM, Patrick Sanan  wrote:
> 
> The manual and dev manual automated builds are failing on master, but I can't 
> reproduce on any of my local builds and the output is now very terse, so it's 
> not obvious to me how to debug. There were some updates recently to the 
> manual make process, which could be related.
> 
> I'd move to re-introduce more of the tex output by default when building the 
> manuals - this clutters the log but overall I'd say would save time (and 
> might be the most efficient way to debug this current issue).

Re: [petsc-dev] Manual builds fail on master

[Smith, Barry F.]

  Satish,

   Where did you find this error message on the web? I couldn't find any 
building of the manual from the Dashboard.


> On Mar 22, 2018, at 11:05 AM, Balay, Satish  wrote:
> 
>>>>>>>>>> 
> petsc@thwomp:/sandbox/petsc/petsc.clone/src/docs/tex/manual$ /usr/bin/make  
> --no-print-directory manual.pdf LOC=/sandbox/petsc/petsc.clone GONULL=
> 
> 
> aTeX Warning: Reference `ch_index' on page 7 undefined on input line 42.
> 
> ) [7] [8] (./acknowltmp.tex
> ! You can't use `macro parameter character #' in horizontal mode.
> l.1 ...++man+manualpages/Mat/MatMPIAdjToSeq-.html#
>  MatMPIAdjToSeq-
> !  ==> Fatal error occurred, no output PDF file produced!
> Transcript written on manual1.log.
> make: *** [manual.pdf] Error 1
> <<<<<<<<<<<
> 
> Perhaps the following fix:
> 
>>>>>>>>> 
> 
> diff --git a/src/mat/impls/adj/mpi/mpiadj.c b/src/mat/impls/adj/mpi/mpiadj.c
> index 5241a8fae9..e8425726d9 100644
> --- a/src/mat/impls/adj/mpi/mpiadj.c
> +++ b/src/mat/impls/adj/mpi/mpiadj.c
> @@ -806,7 +806,7 @@ PETSC_EXTERN PetscErrorCode MatCreate_MPIAdj(Mat B)
> }
> 
> /*@C
> -   MatMPIAdjToSeq- Converts an parallel MPIAdj matrix to complete MPIAdj on 
> each process (needed by sequential preconditioners)
> +   MatMPIAdjToSeq - Converts an parallel MPIAdj matrix to complete MPIAdj on 
> each process (needed by sequential preconditioners)
> 
>Logically Collective on MPI_Comm
> <<<<<<
> 
> Will check if this works..
> 
> Satish
> 
> 
> On Thu, 22 Mar 2018, Smith, Barry F. wrote:
> 
>> 
>>   I'll talk to Satish, it is now unclear (at least to me) where the manual 
>> is actually built and the output stored.
>> 
>> 
>>> On Mar 22, 2018, at 5:17 AM, Patrick Sanan  wrote:
>>> 
>>> The manual and dev manual automated builds are failing on master, but I 
>>> can't reproduce on any of my local builds and the output is now very terse, 
>>> so it's not obvious to me how to debug. There were some updates recently to 
>>> the manual make process, which could be related.
>>> 
>>> I'd move to re-introduce more of the tex output by default when building 
>>> the manuals - this clutters the log but overall I'd say would save time 
>>> (and might be the most efficient way to debug this current issue).
>> 
>> 
> 

Re: [petsc-dev] MatLoad/VecLoad for a file with multiple objects

[Smith, Barry F.]

   The intention is that if one wants to do something "fancy" with storing 
objects to files they use a tool for this (is HDF5 such a tool?) while PETSc 
binary files only provide simple straightforward way to store objects (this is 
similar to the graphics tools in PETSc that also not intended to be 
sophisticated but just provide "quick and dirty" graphics). If you wish to 
contribute clearer documentation on the limitations of the PETSc binary format 
that is great.

   I find "Not a vector next in file" to be an accurate statement. 

   Barry


> On Mar 23, 2018, at 11:39 AM, Vaclav Hapla  wrote:
> 
> Hello
> 
> I'm quite surprised that there can be multiple objects in a single binary 
> file, and they must be loaded in the exact order they are in the file. This 
> behavior is not documented in MatLoad or VecLoad.
> 
> For instance, there can be a matrix A, RHS b and initial guess x0 stored in 
> one file in this order, but you have to call
>  MatLoad(A,viewer); VecLoad(b,viewer); VecLoad(x0,viewer);
> in the same order.
> 
> When you e.g. call VecLoad first, it fails with "Not a vector next in file". 
> This I think is not exactly informative (it says there's no vector but 
> doesn't say that there's something different, actually a matrix).
> 
> I can make a PR documenting this, maybe also improving the error message, if 
> you otherwise find this behavior OK.
> 
> Vaclav

Re: [petsc-dev] TS: -info_exclude and -log_exclude

[Smith, Barry F.]

  I think implementing both would be good.



> On Mar 29, 2018, at 8:05 AM, Lisandro Dalcin  wrote:
> 
> Question for Barry:
> 
> Right now, `-info_exclude ts and `-log_exclude ts` deactivate TS, but
> not DMTS, TSAdapt, and TSTrajectory, and there is no way to deactivate
> them from cmd line. I want to somehow fix this.
> 
> Option 1: Passing `-xxx_exclude ts` deactivates everything TS-related.
> 
> Option 2: We add `-xxx_exclude ` for
> fine-grained deactivation.
> 
> Option 3: We implement both option 1 and option 2.
> 
> So, Barry, what's your preference? The quickest one is just (1) but
> the others are also rather trivial to implement.
> 
> 
> 
> 
> -- 
> Lisandro Dalcin
> 
> Research Scientist
> Computer, Electrical and Mathematical Sciences & Engineering (CEMSE)
> Extreme Computing Research Center (ECRC)
> King Abdullah University of Science and Technology (KAUST)
> http://ecrc.kaust.edu.sa/
> 
> 4700 King Abdullah University of Science and Technology
> al-Khawarizmi Bldg (Bldg 1), Office # 0109
> Thuwal 23955-6900, Kingdom of Saudi Arabia
> http://www.kaust.edu.sa
> 
> Office Phone: +966 12 808-0459

Re: [petsc-dev] upcoming release and testing

[Smith, Barry F.]

   Everyone,

  I think this model is way to labor intense on Satish to keep permanently 
but we'll keep it to get past the release and then rethink the model. Satish is 
also working to decrease the test time so we can hopefully open up more testing 
slots. My dream is each person gets a testing slot for themselves but this is 
not likely in the short term so we may have a small number of people share each 
testing slot.

 Barry


> On Apr 1, 2018, at 11:24 AM, Satish Balay  wrote:
> 
> PETSc developers,
> 
> I've change daily/nightly tests to have 3 tests every day [from the current 2 
> tests]:
> 
> - next
> - next-tmp [can change this test slot as needed]
> - master
> 
> Please note the time/schedule of these tests 
> http://ftp.mcs.anl.gov/pub/petsc/nightlylogs/
> 
> next; run everyday 12am to 7:30am Chicago time
> next-tmp; run everyday 8am to 3:30pm
> master; run weekdays 4pm to 11:30pm
> 
> [I haven't yet added back 'maint' - will do it at some point]
> 
> Plan is to test branches in next-tmp before merge to master so:
> 
> Please do *not* merge any feature branches from next to master without 
> additional testing via next-tmp.
> 
> If you think a branch is ready for merge - let me know - and I'll schedule it 
> into next-tmp
> 
> Also - please look at breakages in master [without a clean master - its 
> difficult to know when feature branches in next/next-tmp are clean. And when 
> feature branches from dirty next are merged to master - master gets more 
> broke]
> 
> I'll try to send follow up emails on master brakages.
> 
> Notes about next-tmp:
> 
> - for now only I have push access to it [so that I can test selected feature 
> branches with it, and reset it as needed]
> 
> - it will be 'latest-master + a few test branches' and updated with 'push -f' 
> [i.e not a continuously updated branch like next]
> 
> Thanks,
> Satish

Re: [petsc-dev] upcoming release and testing

[Smith, Barry F.]

  Satish,

This happens on that one machine where it produces MUCH less precision then 
for any other block size or machine configuration. I investigated a little when 
it first happened and my guess is there some issue with the compiler producing 
"bad" code on this one system.

I hate to have an alt file for a buggy compiler but don't have any other 
suggestion go ahead and add the alt file.

  Barry


> On Apr 1, 2018, at 12:08 PM, Balay, Satish  wrote:
> 
> On Sun, 1 Apr 2018, Satish Balay wrote:
> 
>> I'll try to send follow up emails on master brakages.
> 
> Barry,
> 
> http://ftp.mcs.anl.gov/pub/petsc/nightlylogs/archive/2018/03/31/examples_master_arch-cuda-single_es.log
> 
> not ok diff-ksp_ksp_tests-ex49_1_bs-7
> # 0a1
> # > Norm of residual 0.00159951 iterations 2 bs 7
> 
> This came up on 2018/01/24 - I think - from migrating from old to new 
> testsuite. How to fix? Add alt output file?
> 
> 
> diff --git a/src/ksp/ksp/examples/tests/output/ex49_1_alt.out 
> b/src/ksp/ksp/examples/tests/output/ex49_1_alt.out
> new file mode 100644
> index 000..828d9d9
> --- /dev/null
> +++ b/src/ksp/ksp/examples/tests/output/ex49_1_alt.out
> @@ -0,0 +1 @@
> +Norm of residual 0.00159951 iterations 2 bs 7
> 
> 
> Thanks,
> Satish
> 
> 

[petsc-dev] Issues of limited number of MPI communicators when having many instances of hypre boomerAMG with Moose

[Smith, Barry F.]

   hypre developers,

 Some of the MPI implementations only support around 2000 MPI 
communicators, this can cause difficulties for MOOSE users who have many 
instances of hypre BoomerAMG at the same time. Thus I have some questions for 
you,

For each HYPRE matrix we do a MPI_Comm_dup() to make sure that hypre has a 
unique communicator that won't conflict with PETSc communicators or other hypre 
communicators associated with other matrices.  Do we need to do this? Is hypre 
smart enough that we could pass the same communicator with all the matrices 
(assuming the matrices live on the same communicator)?

It appears that each BoomerAMG instance calls MPI_Comm_create() down in the 
guts (even on one process where the new communicator is the same size as the 
original communicator). Is this necessary? Could you detect when the 
sub-communicator is the same size as the original communicator and then skip 
the MPI_Comm_create()?

   Due to the limitations above MOOSE users are limited to about 1,000 
instances of hypre BoomerAMG at the same time

   Any thoughts on how this limitation could be increased?

Thanks

  Barry

Re: [petsc-dev] [issue1595] Issues of limited number of MPI communicators when having many instances of hypre boomerAMG with Moose

[Smith, Barry F.]

> On Apr 3, 2018, at 1:46 PM, Rob Falgout hypre Tracker 
>  wrote:
> 
> 
> Rob Falgout  added the comment:
> 
> Hi Barry,
> 
> It looks like the only time we call MPI_Comm_create is to build a 
> communicator for the coarsest grid solve using Gaussian elimination.  There 
> are probably alternatives that do not require creating a sub-communicator.

When the sub communicator is of size 1 you can use MPI_COMM_SELF instead of 
creating a new communicator each time.

>  Ulrike or someone else more familiar with the code should comment.
> 
> I don't see a need to do a Comm_dup() before calling hypre.

   If I have 1000 hypre solvers on the same communicator how do I know that 
hypre won't send messages between the different solvers and hence gets messed 
up? In other words how do you handle tags to prevent conflicts between 
different matrices? What communicators do you actually do communication on and 
where do you get them? From the hypre matrix?



> 
> Hope this helps.
> 
> -Rob
> 
> --
> status: unread -> chatting
> 
> 
> hypre Issue Tracker 
> 
> 

Re: [petsc-dev] [issue1595] Issues of limited number of MPI communicators when having many instances of hypre boomerAMG with Moose

[Smith, Barry F.]

> On Apr 3, 2018, at 4:30 PM, Li, Ruipeng  wrote:
> 
> Hi, Fande,
> 
> "For instance, if an application has multiple fields and each field will be 
> solved by a HYPRE solver,  then we end up with  multiple hypre solves. The 
> multiple hypre solves are not run simultaneously, instead, it will be run 
> one-by-one. If we assign the same communicator to the multiple hypre solves, 
> any potential tag conflict?"
> 
> In this scenario, is just the solve phase of BoomerAMG one after another, or 
> also the setup of AMG? Since BoomerAMG takes the communicator from the A 
> matrix, if the entire hypre solve (setup + solve) is serial, there should be 
> no conflicts.

It would be mostly be 

setup on first matrix solve on first matrix
setup on second matrix solve on second matrix
.
solve on first matrix
solve on second matrix
...
setup on first matrix solve on first matrix
setup on second matrix solve on second matrix
.
solve on first matrix
solve on second matrix
...

  Barry


> 
> -Ruipeng
> 
> . -- .- .. .-.. / ..-. .-. --- -- / .-. ..- .. .--. . -. --. / .-.. ..
> Ruipeng Li
> Center for Applied Scientific Computing
> Lawrence Livermore National Laboratory
> P.O. Box 808, L-561
> Livermore, CA 94551
> phone - (925) 422-6037,  email - l...@llnl.gov
> 
> 
> 
> From: Kong, Fande 
> Sent: Tuesday, April 3, 2018 1:34 PM
> To: hypre-support
> Cc: Barry Smith; Derek Gaston; Li, Ruipeng; Osei-Kuffuor, Daniel; For users 
> of the development version of PETSc; Schroder, Jacob B.; tza...@llnl.gov; 
> umy...@llnl.gov; Wang, Lu
> Subject: Re: [issue1595] Issues of limited number of MPI communicators when 
> having many instances of hypre boomerAMG with Moose
> 
> 
> 
> On Tue, Apr 3, 2018 at 2:12 PM, Rob Falgout hypre Tracker 
> mailto:hypre-supp...@llnl.gov>> wrote:
> 
> Rob Falgout mailto:rfalg...@llnl.gov>> added the comment:
> 
> Hi Barry,
> 
> Can you explain the scenario where multiple hypre solves would be working in 
> parallel with the same communicator?  Thanks!
> 
> Hi Rob,
> 
> For instance, if an application has multiple fields and each field will be 
> solved by a HYPRE solver,  then we end up with  multiple hypre solves. The 
> multiple hypre solves are not run simultaneously, instead, it will be run 
> one-by-one. If we assign the same communicator to the multiple hypre solves, 
> any potential tag conflict?
> 
> Fande,
> 
> 
> 
> -Rob
> 
> 
> hypre Issue Tracker mailto:hypre-supp...@llnl.gov>>
> 
> 
> 

Re: [petsc-dev] FAS indentation

[Smith, Barry F.]

   Alp,

 Can you please take a look at this?

  Thanks

Barry


> On Apr 3, 2018, at 4:46 PM, Matthew Knepley  wrote:
> 
> The recent indentation fix (I think) has broken the FAS solver indentation.
> 
>Matt
> 
> -- 
> What most experimenters take for granted before they begin their experiments 
> is infinitely more interesting than any results to which their experiments 
> lead.
> -- Norbert Wiener
> 
> https://www.cse.buffalo.edu/~knepley/

Re: [petsc-dev] Nullspace with right preconditioning

[Smith, Barry F.]

  It may not be broken; it may just be misunderstood or incorrect approach.

  You'll need to explain why and where exactly you think it is broken.

  Barry




> On Apr 4, 2018, at 1:13 PM, Matthew Knepley  wrote:
> 
> This is broken. Did we know this was broken?
> 
>   Thanks,
> 
>  Matt
> 
> -- 
> What most experimenters take for granted before they begin their experiments 
> is infinitely more interesting than any results to which their experiments 
> lead.
> -- Norbert Wiener
> 
> https://www.cse.buffalo.edu/~knepley/

Re: [petsc-dev] nested timers transfers and reductions

[Smith, Barry F.]

> On Apr 6, 2018, at 5:22 AM, Klaij, Christiaan  wrote:
> 
> Hi Barry,
> 
> In our version of the nested timers, we had the functions
> PetscAdminSend, PetscAdminRecv and PetscAdminReduce. So I was
> expecting to find them at the end of the file
> petsc-3.8.4/src/sys/logging/xmllogevent.c but apparently
> you (re)moved them?

   I do not remember why I would have removed these functions. If you can send 
a patch that would return them I would gladly try to get them working with 
PETSc again.

Barry

> 
> We basically used these functions to fill the transfer(GiB/s) and
> reductions/s columns of the log just like we use PetscLogFlops to
> fill the compute(Mflops) column for our registered events.
> 
> How should I do this with the petsc version of the nested timers?
> 
> Chris
> 
> 
> dr. ir. Christiaan Klaij  | Senior Researcher | Research & Development
> MARIN | T +31 317 49 33 44 | mailto:c.kl...@marin.nl | http://www.marin.nl
> 
> MARIN news: 
> http://www.marin.nl/web/News/News-items/Application-of-potential-flow-methods-to-fast-displacement-ships-at-transcritical-speeds-in-shallow-water-1.htm
> 

Re: [petsc-dev] nested timers transfers and reductions

[Smith, Barry F.]

   Hopefully all these problems are resolved with 
https://bitbucket.org/petsc/petsc/pull-requests/925/fix-some-missing-functionality-for-nested/diff

  Barry



> On Apr 9, 2018, at 4:37 AM, Klaij, Christiaan  wrote:
> 
> Hi Barry,
> 
> I've put the functions and fortran bindings in the attached
> files. In our code, we use them as follows:
> 
>  CALL mpi_bcast(value,1,MPI_LOGICAL,iproc,MPI_COMM_WORLD,ier)
>  IF(ier/=0) CALL tracing_abort('ERROR in bcast logical root !')
> 
>  IF (iproc==thisprocess) THEN
> ier = PetscAdminSend(1*(nprocesses-1),MPI_LOGICAL)
>  ELSE
> ier = PetscAdminRecv(1,MPI_LOGICAL)
>  END IF
> 
> I also noted that you changed the treshold from 0.1% to 0.01%, so
> I tried to change it as a user with PetscLogSetThreshold, but
> this function seems not accessible to users (and there's no
> fortran version or manual page).
> 
> Chris
> 
> 
> dr. ir. Christiaan Klaij  | Senior Researcher | Research & Development
> MARIN | T +31 317 49 33 44 | mailto:c.kl...@marin.nl | http://www.marin.nl
> 
> MARIN news: 
> http://www.marin.nl/web/News/News-items/Kom-zaterdag-10-november-naar-de-open-dag-in-Wageningen.htm
> 
> 
> From: Smith, Barry F. 
> Sent: Saturday, April 07, 2018 5:36 PM
> To: Klaij, Christiaan
> Cc: petsc-dev@mcs.anl.gov; Koos Huijssen
> Subject: Re: nested timers transfers and reductions
> 
>> On Apr 6, 2018, at 5:22 AM, Klaij, Christiaan  wrote:
>> 
>> Hi Barry,
>> 
>> In our version of the nested timers, we had the functions
>> PetscAdminSend, PetscAdminRecv and PetscAdminReduce. So I was
>> expecting to find them at the end of the file
>> petsc-3.8.4/src/sys/logging/xmllogevent.c but apparently
>> you (re)moved them?
> 
>   I do not remember why I would have removed these functions. If you can send 
> a patch that would return them I would gladly try to get them working with 
> PETSc again.
> 
>Barry
> 
>> 
>> We basically used these functions to fill the transfer(GiB/s) and
>> reductions/s columns of the log just like we use PetscLogFlops to
>> fill the compute(Mflops) column for our registered events.
>> 
>> How should I do this with the petsc version of the nested timers?
>> 
>> Chris
>> 
>> 
>> dr. ir. Christiaan Klaij  | Senior Researcher | Research & Development
>> MARIN | T +31 317 49 33 44 | mailto:c.kl...@marin.nl | http://www.marin.nl
>> 
>> MARIN news: 
>> http://www.marin.nl/web/News/News-items/Application-of-potential-flow-methods-to-fast-displacement-ships-at-transcritical-speeds-in-shallow-water-1.htm
>> 
> 
> 

Re: [petsc-dev] getting on with the nested timers

[Smith, Barry F.]

   Hopefully all these problems are resolved with 
https://bitbucket.org/petsc/petsc/pull-requests/925/fix-some-missing-functionality-for-nested/diff

  Barry


> On Apr 5, 2018, at 6:31 AM, Klaij, Christiaan  wrote:
> 
> Matt,
> 
> I guess you meant to send this link:
> 
> http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Viewer/PetscViewerPushFormat.html
> 
> If so, it doesn't say anything on XML either. But after some
> trial and error I found the format to
> be "PETSC_VIEWER_ASCII_XML", so problem solved.
> 
> Summarizing the problem from a user perspective:
> 
> 1) The manual page of PetscLogView mentions PetscViewerASCIIOpen,
> which mentions PetscViewerPushFormat, I should have figured that
> out by myself, sorry.
> 
> 2) The manual page of PetscLogView mentions PetscLogNestedBegin,
> which doesn't have a manual entry.
> 
> 3) The manual page of PetscViewerPushFormat doesn't mention the
> format PETSC_VIEWER_ASCII_XML, although it is available.
> 
> Chris
> 
> dr. ir. Christiaan Klaij | Senior Researcher | Research & Development
> MARIN | T +31 317 49 33 44 | c.kl...@marin.nl | www.marin.nl
> 
>
> MARIN news: Comfort draft for semi-submersible yachts
> From: Matthew Knepley 
> Sent: Thursday, April 05, 2018 12:50 PM
> To: Klaij, Christiaan
> Cc: Smith, Barry F.; petsc-dev@mcs.anl.gov; Koos Huijssen
> Subject: Re: [petsc-dev] getting on with the nested timers
>  
> On Thu, Apr 5, 2018 at 6:15 AM, Klaij, Christiaan  wrote:
> Barry,
> 
> I'm still trying to use petsc's nested logging with our code. I
> have it working for 3.8.3 with the command line option, for
> example:
> 
> mpirun -n 3 ./refresco -log_view :performance.xml:ascii_xml
> 
> This gives the expected performance.xml file. How to get the same
> without command line options? So far I have:
> 
> CALL PetscLogDefaultBegin(ierr)
> ...
> CALL PetscViewerASCIIOpen(MPI_COMM_WORLD,'performance.xml',viewer,ierr)
> CALL PetscLogView(viewer,ierr)
> 
> which gives the regular log in the performance.xml file.
> 
> I guess I should replace PetscLogDefaultBegin by
> PetscLogNestedBegin as stated in the manual page of PetscLogView.
> 
> But there's no manual page for PetscLogNestedBegin and I didn't
> find anything like PetscViewerASCIIOpenXML either.
> 
> http://www.mcs.anl.gov/petsc/petsc-master/docs/manualpages/Profiling/PetscLogView.html
> 
> You need to push the ASCII_XML format before viewing.
> 
>   Thanks,
> 
>  Matt
>  
> Chris
> 
> 
> dr. ir. Christiaan Klaij  | Senior Researcher | Research & Development
> MARIN | T +31 317 49 33 44 | mailto:c.kl...@marin.nl | http://www.marin.nl
> 
> MARIN news: 
> http://www.marin.nl/web/News/News-items/Kom-zaterdag-10-november-naar-de-open-dag-in-Wageningen.htm
> 
> 
> 
> 
> -- 
> What most experimenters take for granted before they begin their experiments 
> is infinitely more interesting than any results to which their experiments 
> lead.
> -- Norbert Wiener
> 
> https://www.cse.buffalo.edu/~knepley/

Re: [petsc-dev] make check fortran warnings for complex single 64idx

[Smith, Barry F.]

   We have to live with these warnings.

Barry


> On Apr 11, 2018, at 10:17 AM, Vaclav Hapla  wrote:
> 
> Hello
> 
> I just compiled current master with
>'--with-precision=single',
>'--with-scalar-type=complex',
>'--with-64-bit-indices',
> 
> configure and make work, but 'make check' throws some warnings regarding 
> src/snes/examples/tutorials/ex5f.F90, see below.
> 
> mpifort —version
> says
> GNU Fortran (Ubuntu 5.4.0-6ubuntu1~16.04.9) 5.4.0 20160609
> 
> I'm not much into Fortran - can you see what's going wrong?
> 
> Vaclav
> 
> 
> 
> C/C++ example src/snes/examples/tutorials/ex19 run successfully with 2 MPI 
> processes
> ***Error detected during compile or link!***
> See http://www.mcs.anl.gov/petsc/documentation/faq.html
> /home/vhapla/petsc/src/snes/examples/tutorials ex5f
> *
> mpifort -c -fPIC -Wall -ffree-line-length-0 -Wno-unused-dummy-argument -g
> -I/home/vhapla/petsc/include 
> -I/home/vhapla/petsc/arch-linux-gcc-complex-single-64idx/include 
> -I/home/vhapla/include-o ex5f.o ex5f.F90
> ex5f.F90:289:17:
> 
>  temp = (min(j-1,my-j))*hy
> 1
> Warning: Possible change of value in conversion from INTEGER(8) to REAL(4) at 
> (1) [-Wconversion]
> ex5f.F90:294:43:
> 
>   x(i,j) = temp1 * sqrt(min(min(i-1,mx-i)*hx,(temp)))
>   1
> Warning: Possible change of value in conversion from INTEGER(8) to REAL(4) at 
> (1) [-Wconversion]
> ex5f.h:29:21:
> 
>   common /params/ lambda,mx,my
> 1
> Warning: Padding of 4 bytes required before ‘mx’ in COMMON ‘params’ at (1); 
> reorder elements or use -fno-align-commons
> ex5f.h:29:21:
> 
>   common /params/ lambda,mx,my
> 1
> Warning: Padding of 4 bytes required before ‘mx’ in COMMON ‘params’ at (1); 
> reorder elements or use -fno-align-commons
> ex5f.h:29:21:
> 
>   common /params/ lambda,mx,my
> 1
> Warning: Padding of 4 bytes required before ‘mx’ in COMMON ‘params’ at (1); 
> reorder elements or use -fno-align-commons
> ex5f.h:29:21:
> 
>   common /params/ lambda,mx,my
> 1
> Warning: Padding of 4 bytes required before ‘mx’ in COMMON ‘params’ at (1); 
> reorder elements or use -fno-align-commons
> ex5f.h:29:21:
> 
>   common /params/ lambda,mx,my
> 1
> Warning: Padding of 4 bytes required before ‘mx’ in COMMON ‘params’ at (1); 
> reorder elements or use -fno-align-commons
> 

Re: [petsc-dev] [petsc-users] PetscPrintf

[Smith, Barry F.]

> On Apr 12, 2018, at 3:59 AM, Patrick Sanan  wrote:
> 
> I also happened to stumble across this yesterday. Is the length restriction 
> for the default printer (l assume from the array of 8*1024 chars in 
> PetscVFPrintfDefault() ) intended behavior to be documented, or a bug to be 
> fixed?

 You could call it either. My problems are 

1) that given a format string I don't know in advance how much work space is 
needed so cannot accurately malloc, for sure, enough space

2) since this can be called in an error handler I really don't want it calling 
malloc().

   Hence it lives in this limbo. I don't even know a way to add a good error 
checking that detects if the buffer is long enough. All in all it is bad ugly 
code, any suggestions on improvements would be appreciated.

   Barry

> 
> 2018-04-12 2:16 GMT+02:00 Rongliang Chen :
> Thanks Barry. I found petsc-3.6 and older versions did not have this 
> restriction.
> 
> Best,
> Rongliang
> 
> 
> On 04/12/2018 07:22 AM, Smith, Barry F. wrote:
>Yes, PetscPrintf() and related functions have a maximum string length of 
> about 8000 characters.
> 
> Barry
> 
> 
> On Apr 11, 2018, at 6:17 PM, Rongliang Chen  wrote:
> 
> Dear All,
> 
> 
> When I tried to print a long string using PetscPrintf() I found that it 
> truncated the string. Attached is a simple example for this (run with single 
> processor). I used PetscPrintf() and printf() to print the same string and 
> the printf() seems OK. I am using petsc-3.8.4.
> 
> 
> Best,
> 
> Rongliang
> 
> 
> 
> 
> 

Re: [petsc-dev] vector inner products

[Smith, Barry F.]

  I don't know if we are ready for this dramatic change. I wouldn't do it until 
there was a clear need (not just potential future usage) that would be 
cumbersome to do without this generality. I am a little scared of this without 
demonstrated general need.

  I would start by having an abstract inner product object (which includes a 
norm function) that has the standard l2 implementation, then another 
implementation that is based on passing in a matrix, maybe one based on passing 
in a vector.  Then each solver would have a XXXSetInnerProduct() while 
defaulting to l2.

  Barry


> On Apr 12, 2018, at 12:21 PM, Munson, Todd  wrote:
> 
> 
> There is a bit of code in TAO that allows the user to change the norm to 
> a matrix norm.  This was introduced to get some mesh independent 
> behavior in one example (tao/examples/tutorials/ex3.c).  That 
> norm, however, does not propagate down into the KSP methods
> and is only used for testing convergence of the nonlinear
> problem.
> 
> A few questions then:  Is similar functionality needed in SNES?  Are 
> TAO and SNES even the right place for this functionality?  Should 
> it belong to the Vector class so that you can change the inner 
> products and have all the KSP methods (hopefully) work 
> correctly?
> 
> Note: that this discussion brings us to the brink of supporting an 
> optimize-then-discretize approach.  I am not convinced we should 
> go down that rabbit hole.
> 
> Thanks, Todd.
> 

Re: [petsc-dev] [petsc-users] PetscPrintf

[Smith, Barry F.]

> Is the postprocessing of output in PetscVSNPrintf really necessary?

PetscVSNPrintf does two things

1) handles %D to have a common way of printing PetscInt regardless of if they 
are 32 or 64 bit integers

2) insures that all floating point numbers printed have a . in them so 
Petscdiff can accurately determine what is a 
floating point number in output (standard C formats don't put the . in floating 
point numbers like 1. that don't need them. I could not find 
any decent numerical diff for PETSc output to use in place of our cumbersome 
home-brew thing.

If there are alternative ways of handling 1) and 2) then PetscVSNPrintf() and 
its cumbersomeness are not needed.

Barry

Note that PetscVSNPrintf() does not handle printing of PetscReal, we have to 
manually put a caste of (double) in the calling sequence (for __float128). I 
tried to add this support with %G but gave up because it was a rat hole.





> On Apr 12, 2018, at 8:22 PM, Jed Brown  wrote:
> 
> "Smith, Barry F."  writes:
> 
>>> On Apr 12, 2018, at 3:59 AM, Patrick Sanan  wrote:
>>> 
>>> I also happened to stumble across this yesterday. Is the length restriction 
>>> for the default printer (l assume from the array of 8*1024 chars in 
>>> PetscVFPrintfDefault() ) intended behavior to be documented, or a bug to be 
>>> fixed?
>> 
>> You could call it either. My problems are 
>> 
>> 1) that given a format string I don't know in advance how much work space is 
>> needed so cannot accurately malloc, for sure, enough space
>> 
>> 2) since this can be called in an error handler I really don't want it 
>> calling malloc().
>> 
>>   Hence it lives in this limbo. I don't even know a way to add a good error 
>> checking that detects if the buffer is long enough. All in all it is bad 
>> ugly code, any suggestions on improvements would be appreciated.
> 
> Is the postprocessing of output in PetscVSNPrintf really necessary?
> Without it, you would call vfprintf instead of vsnprintf followed by
> fprintf("%s", string) [1].
> 
> 
> [1] fputs would be preferable.
> 
>> 
>>   Barry
>> 
>>> 
>>> 2018-04-12 2:16 GMT+02:00 Rongliang Chen :
>>> Thanks Barry. I found petsc-3.6 and older versions did not have this 
>>> restriction.
>>> 
>>> Best,
>>> Rongliang
>>> 
>>> 
>>> On 04/12/2018 07:22 AM, Smith, Barry F. wrote:
>>>   Yes, PetscPrintf() and related functions have a maximum string length of 
>>> about 8000 characters.
>>> 
>>>Barry
>>> 
>>> 
>>> On Apr 11, 2018, at 6:17 PM, Rongliang Chen  
>>> wrote:
>>> 
>>> Dear All,
>>> 
>>> 
>>> When I tried to print a long string using PetscPrintf() I found that it 
>>> truncated the string. Attached is a simple example for this (run with 
>>> single processor). I used PetscPrintf() and printf() to print the same 
>>> string and the printf() seems OK. I am using petsc-3.8.4.
>>> 
>>> 
>>> Best,
>>> 
>>> Rongliang
>>> 
>>> 

Re: [petsc-dev] [petsc-users] PetscPrintf

[Smith, Barry F.]

   To me the difficulty is knowing how large an array to allocate (in both 
PetscVSNPrintf() and PetscVFPrintfDefault()). The macro 
PETSC_MAX_LENGTH_FORMAT() is used in PetscVSNPrintf() but it has nothing to do 
with reality.  One could walk through the formats (and their string values) to 
get a much better estimate for total length of buffer needed. 

Maybe a routine something like

   va_list Argp;
va_start(Argp,format);
ierr = PetscDetermineOutputLength(format,Argp,size_t 
*neededlength);CHKERRQ(ierr);
PetscMalloc(neededlength,&buffer);


But I am not convinced we can't just have an upper bound on the possible format 
length (so long as we can error out properly if the user requests a something 
that doesn't fit).


Barry

Note: there are horrible other buffer size code fragments such as 

next->size = -1;
while ((PetscInt)fullLength >= next->size) {
  next->size = fullLength+1;

  ierr = PetscMalloc1(next->size, &next->string);CHKERRQ(ierr);
  va_start(Argp,format);
  ierr = PetscMemzero(next->string,next->size);CHKERRQ(ierr);
  ierr = PetscVSNPrintf(next->string,next->size,format, 
&fullLength,Argp);CHKERRQ(ierr);
  va_end(Argp);
}

that seems to try (very badly) to allocate larger and larger buffers (without 
freeing the previous?) until it has enough room for the result ? Any fix should 
resolve this as well.

   Maybe two versions of PetscVSNPrintf() one that takes a buffer (and respects 
it) and one that allocates a buffer large enough.




> On Apr 15, 2018, at 5:41 AM, Patrick Sanan  wrote:
> 
> How about the logic of this analysis?
> 
> 1. We are trying to use the same functions (in particular, PetscVFPrintf) for 
> two purposes:
>a. printing error messages (don't want to malloc)
>b. use by public API printing functions (don't want length restrictions)
> 
> 2. Right now, PetscVFPrintf works fine for a but not for b. We could make it 
> work for b and not for a by malloc'ing a longer string.
> 
> 3. Printing from error handlers happens through PetscErrorPrintf (default : 
> http://www.mcs.anl.gov/petsc/petsc-dev/src/sys/error/errtrace.c.html#PetscErrorPrintfDefault
>  ), so if there's a special requirement for printing error messages, we can 
> impose it here.
> 
> A solution could then be something which skips the malloc only when printing 
> an error, e.g.
> 
> 1. Add an argument to PetscVFPrintf (say "PetscBool noMalloc") [1]
> 2. API (PetscPrintf(), etc.) functions use noMalloc = PETSC_FALSE
> 3. error functions (PetscErrorPrintf() ) functions use noMalloc = PETSC_TRUE 
> 
> 
> [1] And probably do the same thing in PetscVSNPrintf since, as Dr. Zhang 
> pointed out, this could also call malloc while handling an error, if the 
> string was long enough
> 
> 
> 
> 
> 2018-04-13 15:59 GMT+02:00 Junchao Zhang :
> On Thu, Apr 12, 2018 at 9:48 AM, Smith, Barry F.  wrote:
> 
> 
> > On Apr 12, 2018, at 3:59 AM, Patrick Sanan  wrote:
> >
> > I also happened to stumble across this yesterday. Is the length restriction 
> > for the default printer (l assume from the array of 8*1024 chars in 
> > PetscVFPrintfDefault() ) intended behavior to be documented, or a bug to be 
> > fixed?
> 
>  You could call it either. My problems are
> 
> 1) that given a format string I don't know in advance how much work space is 
> needed so cannot accurately malloc, for sure, enough space
> 
> 2) since this can be called in an error handler I really don't want it 
> calling malloc().
> PetscVSNPrintf does still contain a malloc "122  ierr  = 
> PetscMalloc1(oldLength, &newformat);CHKERRQ(ierr);"
> Also, vsnprintf returns "the number of characters that would have been 
> written if n had been sufficiently large". I don't know why you void'ed it.
> We can not make the 8K chars a requirement since users don't know how many 
> chars they want to print upfront.
> Anyway, crash is better than silent errors. 
> 
>Hence it lives in this limbo. I don't even know a way to add a good error 
> checking that detects if the buffer is long enough. All in all it is bad ugly 
> code, any suggestions on improvements would be appreciated.
> 
>Barry
> 
> >
> > 2018-04-12 2:16 GMT+02:00 Rongliang Chen :
> > Thanks Barry. I found petsc-3.6 and older versions did not have this 
> > restriction.
> >
> > Best,
> > Rongliang
> >
> >
> > On 04/12/2018 07:22 AM, Smith, Barry F. wrote:
> >Yes, PetscPrintf() and related functions have a maximum string length of 
> > about 8000 characters.
> >
> > Barry
> >
> >
> > On Apr 11, 2018, at 6:17 PM, Rongliang Chen  
> > wrote:
> >
> > Dear All,
> >
> >
> > When I tried to print a long string using PetscPrintf() I found that it 
> > truncated the string. Attached is a simple example for this (run with 
> > single processor). I used PetscPrintf() and printf() to print the same 
> > string and the printf() seems OK. I am using petsc-3.8.4.
> >
> >
> > Best,
> >
> > Rongliang
> >
> > 
> >
> >
> >
> 
> 
> 

Re: [petsc-dev] [petsc-users] PetscPrintf

[Smith, Barry F.]

  Very small step in the correct direction 
https://bitbucket.org/petsc/petsc/pull-requests/931/barry-bugfix-petscformatconvert/diff


> On Apr 15, 2018, at 5:41 AM, Patrick Sanan  wrote:
> 
> How about the logic of this analysis?
> 
> 1. We are trying to use the same functions (in particular, PetscVFPrintf) for 
> two purposes:
>a. printing error messages (don't want to malloc)
>b. use by public API printing functions (don't want length restrictions)
> 
> 2. Right now, PetscVFPrintf works fine for a but not for b. We could make it 
> work for b and not for a by malloc'ing a longer string.
> 
> 3. Printing from error handlers happens through PetscErrorPrintf (default : 
> http://www.mcs.anl.gov/petsc/petsc-dev/src/sys/error/errtrace.c.html#PetscErrorPrintfDefault
>  ), so if there's a special requirement for printing error messages, we can 
> impose it here.
> 
> A solution could then be something which skips the malloc only when printing 
> an error, e.g.
> 
> 1. Add an argument to PetscVFPrintf (say "PetscBool noMalloc") [1]
> 2. API (PetscPrintf(), etc.) functions use noMalloc = PETSC_FALSE
> 3. error functions (PetscErrorPrintf() ) functions use noMalloc = PETSC_TRUE 
> 
> 
> [1] And probably do the same thing in PetscVSNPrintf since, as Dr. Zhang 
> pointed out, this could also call malloc while handling an error, if the 
> string was long enough
> 
> 
> 
> 
> 2018-04-13 15:59 GMT+02:00 Junchao Zhang :
> On Thu, Apr 12, 2018 at 9:48 AM, Smith, Barry F.  wrote:
> 
> 
> > On Apr 12, 2018, at 3:59 AM, Patrick Sanan  wrote:
> >
> > I also happened to stumble across this yesterday. Is the length restriction 
> > for the default printer (l assume from the array of 8*1024 chars in 
> > PetscVFPrintfDefault() ) intended behavior to be documented, or a bug to be 
> > fixed?
> 
>  You could call it either. My problems are
> 
> 1) that given a format string I don't know in advance how much work space is 
> needed so cannot accurately malloc, for sure, enough space
> 
> 2) since this can be called in an error handler I really don't want it 
> calling malloc().
> PetscVSNPrintf does still contain a malloc "122  ierr  = 
> PetscMalloc1(oldLength, &newformat);CHKERRQ(ierr);"
> Also, vsnprintf returns "the number of characters that would have been 
> written if n had been sufficiently large". I don't know why you void'ed it.
> We can not make the 8K chars a requirement since users don't know how many 
> chars they want to print upfront.
> Anyway, crash is better than silent errors. 
> 
>Hence it lives in this limbo. I don't even know a way to add a good error 
> checking that detects if the buffer is long enough. All in all it is bad ugly 
> code, any suggestions on improvements would be appreciated.
> 
>Barry
> 
> >
> > 2018-04-12 2:16 GMT+02:00 Rongliang Chen :
> > Thanks Barry. I found petsc-3.6 and older versions did not have this 
> > restriction.
> >
> > Best,
> > Rongliang
> >
> >
> > On 04/12/2018 07:22 AM, Smith, Barry F. wrote:
> >Yes, PetscPrintf() and related functions have a maximum string length of 
> > about 8000 characters.
> >
> > Barry
> >
> >
> > On Apr 11, 2018, at 6:17 PM, Rongliang Chen  
> > wrote:
> >
> > Dear All,
> >
> >
> > When I tried to print a long string using PetscPrintf() I found that it 
> > truncated the string. Attached is a simple example for this (run with 
> > single processor). I used PetscPrintf() and printf() to print the same 
> > string and the printf() seems OK. I am using petsc-3.8.4.
> >
> >
> > Best,
> >
> > Rongliang
> >
> > 
> >
> >
> >
> 
> 
> 

Re: [petsc-dev] Default dimension of a DM

[Smith, Barry F.]

  Seems ok to me.

  Barry


> On Apr 16, 2018, at 9:03 AM, Matthew Knepley  wrote:
> 
> I am fine with this change.
> 
>Matt
> 
> On Mon, Apr 16, 2018 at 9:19 AM, Stefano Zampini  
> wrote:
> Maybe this is silly, but why the default dimenson of a DM is 0; why not -1?
> A DM of dimension 0 is a point, while a DM of dimension -1 is a DM in a not 
> yet complete setup state.
> -- 
> Stefano
> 
> 
> 
> -- 
> What most experimenters take for granted before they begin their experiments 
> is infinitely more interesting than any results to which their experiments 
> lead.
> -- Norbert Wiener
> 
> https://www.cse.buffalo.edu/~knepley/

Re: [petsc-dev] Matlab interface issue

[Smith, Barry F.]

Hmm, I tried on my Mac and had the same problem. But then I upgraded from 
Matlab 2017A to 2018A and boom the problem went away and it was able to build 
the socket mex files without a problem.

So my conclusion is that we shouldn't change anything in the PETSc makefiles?

Barry


> On Apr 16, 2018, at 6:41 AM, Vaclav Hapla  wrote:
> 
> Hello
> 
> I need to compile PETSc with MATLAB interface on Mac OS.
> 
> First thing is that Mex supports only Xcode or Intel compilers, not GCC. So I 
> compiled PETSc with Xcode, using options
>   --with-cc="/usr/bin/xcrun -sdk macosx10.13 clang" --with-matlab 
> --with-matlab-arch=maci64 --with-matlab-engine --with-mpi=0
> 
> But the target "matlabcodes" (in 
> src/sys/classes/viewer/impls/socket/matlab/makefile) was failing with
> Undefined symbols for architecture x86_64:
>   "_main", referenced from:
>  implicit entry/start for main executable
> ld: symbol(s) not found for architecture x86_64
> 
> After some investigation I found out the problem lies in specifying 
> environment variables to the mex compiler. Mex has some predefined values of 
> these variables and fails if we override them. According to MATLAB 
> documentation, it should be possible to just append what we need, e.g.
>   LDFLAGS='${LDFLAGS} ${PETSC_EXTERNAL_LIB_BASIC}'
> but this apparently does not work as excepted, some predefined options are 
> removed this way. I was debugging that by adding -v to ${MATLAB_MEX} calls in 
> the makefile. It seems the injected LDFLAGS is interpreted _after_ something 
> more is added to LDFLAGS by MEX (stupid).
> 
> What works for me is specifying the options directly do ${MATLAB_MEX}. This 
> is also tricky, though, as ${MATLAB_MEX} does not accept all options, e.g. 
> -Wl,-rpath. So I ended up with replacing
>   GCC='${CC}' CC='${PCC}' CFLAGS='${COPTFLAGS} ${CC_FLAGS} ${CCPPFLAGS}' 
> LDFLAGS='${PETSC_EXTERNAL_LIB_BASIC}'
> by
>   ${PETSC_CC_INCLUDES} -L${PETSC_DIR}/${PETSC_ARCH}/lib ${PETSC_LIB_BASIC}
> in src/sys/classes/viewer/impls/socket/matlab/makefile. See the attached 
> patch. This at least compiled.
> 
> 
> Do you think this could break anything? Do you have some better idea?
> 
> BTW It would be better if the matlabcodes target could be turned off by 
> configure, if one is interested only in using MATLAB Engine and wants to 
> avoid problems with the mex compiler.
> 
> Thanks,
> Vaclav
> 

Re: [petsc-dev] KSP accepts mismatched sizes between Mat and PC

[Smith, Barry F.]

Normally they need to be the same, but not always. For KSPLSQR the A matrix may 
be rectangular but the B matrix is A^TA which is square. This special case is 
why we don't do error checking.

Barry


> On Apr 16, 2018, at 1:04 PM, Alp Dener  wrote:
> 
> Hi Barry,
> 
> While I was working on some active-set related stuff in TAO, I discovered 
> that I can call KSPSetOperators() with differently sized Mat and PC objects 
> without getting any errors or warnings about the size mismatch. Note that my 
> Mat object is explicit, but the PC object is a MATSHELL. Is this intended 
> behavior, or is it a bug? 
> 
> —
> Alp Dener

Re: [petsc-dev] [petsc-users] PetscPrintf

[Smith, Barry F.]

  Rongliang,

   I have prepared a fix for this in the branch barry/bugfix-petscformatconvert 
which comes from the master branch of PETSc.

   Unfortunately it requires many changes so cannot be back ported to previous 
releases.

   Thanks for reporting the problem,

Barry


> On Apr 11, 2018, at 6:17 PM, Rongliang Chen  wrote:
> 
> Dear All,
> 
> 
> When I tried to print a long string using PetscPrintf() I found that it 
> truncated the string. Attached is a simple example for this (run with single 
> processor). I used PetscPrintf() and printf() to print the same string and 
> the printf() seems OK. I am using petsc-3.8.4.
> 
> 
> Best,
> 
> Rongliang
> 
> 

Re: [petsc-dev] Matlab interface issue

[Smith, Barry F.]

   Please try the branch barry/feature-matlab-socket/maint  and 
--with-matlab-socket=0 and let me know if it works.

   Barry


> On Apr 17, 2018, at 3:16 AM, Vaclav Hapla  wrote:
> 
> OK. But definitely it's tricky that you try to inject all PETSc compiler 
> options to the mex wrapper which is by documentation only tested with Xcode 
> and Intel and it's always set up just for one compiler (see mex -setup).
> 
> Hence, I would at least vote for config option to turn off this socket mex 
> stuff to bypass potential problems if one doesn't need it. Like 
> --with-matlab-socket=0 or so which would just deactivate the matlabbin 
> target. Or behave like that if --with-matlab-engine=1 and --with-matlab=0 
> (this currently fails with "Did not find package MATLAB needed by 
> MatlabEngine" which is confusing anyway as it's not a standalone product).
> 
> Thanks,
> Vaclav
> 
>> 16. 4. 2018 v 20:30, Smith, Barry F. :
>> 
>> Hmm, I tried on my Mac and had the same problem. But then I upgraded from 
>> Matlab 2017A to 2018A and boom the problem went away and it was able to 
>> build the socket mex files without a problem.
>> 
>> So my conclusion is that we shouldn't change anything in the PETSc makefiles?
>> 
>>Barry
>> 
>> 
>>> On Apr 16, 2018, at 6:41 AM, Vaclav Hapla  wrote:
>>> 
>>> Hello
>>> 
>>> I need to compile PETSc with MATLAB interface on Mac OS.
>>> 
>>> First thing is that Mex supports only Xcode or Intel compilers, not GCC. So 
>>> I compiled PETSc with Xcode, using options
>>>  --with-cc="/usr/bin/xcrun -sdk macosx10.13 clang" --with-matlab 
>>> --with-matlab-arch=maci64 --with-matlab-engine --with-mpi=0
>>> 
>>> But the target "matlabcodes" (in 
>>> src/sys/classes/viewer/impls/socket/matlab/makefile) was failing with
>>> Undefined symbols for architecture x86_64:
>>>  "_main", referenced from:
>>> implicit entry/start for main executable
>>> ld: symbol(s) not found for architecture x86_64
>>> 
>>> After some investigation I found out the problem lies in specifying 
>>> environment variables to the mex compiler. Mex has some predefined values 
>>> of these variables and fails if we override them. According to MATLAB 
>>> documentation, it should be possible to just append what we need, e.g.
>>>  LDFLAGS='${LDFLAGS} ${PETSC_EXTERNAL_LIB_BASIC}'
>>> but this apparently does not work as excepted, some predefined options are 
>>> removed this way. I was debugging that by adding -v to ${MATLAB_MEX} calls 
>>> in the makefile. It seems the injected LDFLAGS is interpreted _after_ 
>>> something more is added to LDFLAGS by MEX (stupid).
>>> 
>>> What works for me is specifying the options directly do ${MATLAB_MEX}. This 
>>> is also tricky, though, as ${MATLAB_MEX} does not accept all options, e.g. 
>>> -Wl,-rpath. So I ended up with replacing
>>>  GCC='${CC}' CC='${PCC}' CFLAGS='${COPTFLAGS} ${CC_FLAGS} ${CCPPFLAGS}' 
>>> LDFLAGS='${PETSC_EXTERNAL_LIB_BASIC}'
>>> by
>>>  ${PETSC_CC_INCLUDES} -L${PETSC_DIR}/${PETSC_ARCH}/lib ${PETSC_LIB_BASIC}
>>> in src/sys/classes/viewer/impls/socket/matlab/makefile. See the attached 
>>> patch. This at least compiled.
>>> 
>>> 
>>> Do you think this could break anything? Do you have some better idea?
>>> 
>>> BTW It would be better if the matlabcodes target could be turned off by 
>>> configure, if one is interested only in using MATLAB Engine and wants to 
>>> avoid problems with the mex compiler.
>>> 
>>> Thanks,
>>> Vaclav
>>> 
>> 
> 

Re: [petsc-dev] Vec set a block of values

[Smith, Barry F.]

   When setting values into matrices and vectors we consider the "extra" 
overhead of needing to pass in the indices for all the values (instead of being 
able to set an arbitrary block of values without using indices for each one) to 
be a minimal overhead that we can live with. 

   Barry


> On Apr 20, 2018, at 3:33 PM, Junchao Zhang  wrote:
> 
> 
> On Fri, Apr 20, 2018 at 3:18 PM, Matthew Knepley  wrote:
> On Fri, Apr 20, 2018 at 4:10 PM, Junchao Zhang  wrote:
> To pad a vector, i.e., copy a vector to a new one, I have to call 
> VecSetValue(newb,1,&idx,...) for each element. But to be efficient, what I 
> really needs is to set a block of values in one call. It looks PETSc does not 
> have a routine for that(?). I looked at VecSetValuesBlocked, but it looks it 
> is not for that purpose.
> Should we have something like VecSetValuesBlock(Vec v,PetscInt i,PetscInt 
> cnt,PetscScalar *value, InsertMode mode) to set cnt values starting at index 
> i?
> 
> Use VecGetArray().
> Did you mean VecGetArray b and newb, do a memcpy from b to new and then 
> restore them? If yes, it does not work since some of the values I want to set 
> might be remote.
> E.g, I have 4 processors. b's size is 181 and is distributed as 46, 45,45,45, 
> newb is distributed as 48,45,45,45 to match a matrix of block size 3.
>  
> 
>   Matt
>  
> --Junchao Zhang
> 
> 
> 
> -- 
> What most experimenters take for granted before they begin their experiments 
> is infinitely more interesting than any results to which their experiments 
> lead.
> -- Norbert Wiener
> 
> https://www.cse.buffalo.edu/~knepley/
> 

Re: [petsc-dev] Vec set a block of values

[Smith, Barry F.]

  Setting large contiguous blocks of values is not a common use case. In finite 
elements the values are not contiguous.

> On Apr 20, 2018, at 3:45 PM, Zhang, Junchao  wrote:
> 
> I agree the extra overhead can be small, but users are forced to write a loop 
> where one single line gives the best.
> 
> --Junchao Zhang
> 
> On Fri, Apr 20, 2018 at 3:36 PM, Smith, Barry F.  wrote:
> 
>When setting values into matrices and vectors we consider the "extra" 
> overhead of needing to pass in the indices for all the values (instead of 
> being able to set an arbitrary block of values without using indices for each 
> one) to be a minimal overhead that we can live with. 
> 
>Barry
> 
> 
> > On Apr 20, 2018, at 3:33 PM, Junchao Zhang  wrote:
> > 
> > 
> > On Fri, Apr 20, 2018 at 3:18 PM, Matthew Knepley  wrote:
> > On Fri, Apr 20, 2018 at 4:10 PM, Junchao Zhang  wrote:
> > To pad a vector, i.e., copy a vector to a new one, I have to call 
> > VecSetValue(newb,1,&idx,...) for each element. But to be efficient, what I 
> > really needs is to set a block of values in one call. It looks PETSc does 
> > not have a routine for that(?). I looked at VecSetValuesBlocked, but it 
> > looks it is not for that purpose.
> > Should we have something like VecSetValuesBlock(Vec v,PetscInt i,PetscInt 
> > cnt,PetscScalar *value, InsertMode mode) to set cnt values starting at 
> > index i?
> > 
> > Use VecGetArray().
> > Did you mean VecGetArray b and newb, do a memcpy from b to new and then 
> > restore them? If yes, it does not work since some of the values I want to 
> > set might be remote.
> > E.g, I have 4 processors. b's size is 181 and is distributed as 46, 
> > 45,45,45, newb is distributed as 48,45,45,45 to match a matrix of block 
> > size 3.
> >  
> > 
> >   Matt
> >  
> > --Junchao Zhang
> > 
> > 
> > 
> > -- 
> > What most experimenters take for granted before they begin their 
> > experiments is infinitely more interesting than any results to which their 
> > experiments lead.
> > -- Norbert Wiener
> > 
> > https://www.cse.buffalo.edu/~knepley/
> > 
> 
> 

Re: [petsc-dev] Vec set a block of values

[Smith, Barry F.]

  I would just use VecSetValues() since almost all values are local it will be 
scalable and a little extra time in setting values is not a big deal for this 
test code.

   Barry


> On Apr 20, 2018, at 4:09 PM, Jed Brown  wrote:
> 
> Junchao Zhang  writes:
> 
>> VecScatter is too heavy (in both coding and runtime) for this simple task.
>> I just want to pad a vector loaded from a PetscViewer to match an MPIBAIJ
>> matrix. Thus the majority is memcpy, with few neighborhood off-processor
>> puts.
> 
> At what address do those puts go, how do you avoid race conditions from
> multiple processors having overlapping neighborhoods, and how does the
> recipient know that the put has completed?  Just use VecScatter.  It
> could be optimized to recognize contiguous runs above a certain size and
> convert to memcpy.
> 
>> --Junchao Zhang
>> 
>> On Fri, Apr 20, 2018 at 3:57 PM, Jed Brown  wrote:
>> 
>>> Junchao, If you need to access off-process values and put them into a
>>> new vector, you should use VecScatter.
>>> 
>>> "Smith, Barry F."  writes:
>>> 
>>>>  Setting large contiguous blocks of values is not a common use case. In
>>> finite elements the values are not contiguous.
>>>> 
>>>>> On Apr 20, 2018, at 3:45 PM, Zhang, Junchao 
>>> wrote:
>>>>> 
>>>>> I agree the extra overhead can be small, but users are forced to write
>>> a loop where one single line gives the best.
>>>>> 
>>>>> --Junchao Zhang
>>>>> 
>>>>> On Fri, Apr 20, 2018 at 3:36 PM, Smith, Barry F. 
>>> wrote:
>>>>> 
>>>>>   When setting values into matrices and vectors we consider the
>>> "extra" overhead of needing to pass in the indices for all the values
>>> (instead of being able to set an arbitrary block of values without using
>>> indices for each one) to be a minimal overhead that we can live with.
>>>>> 
>>>>>   Barry
>>>>> 
>>>>> 
>>>>>> On Apr 20, 2018, at 3:33 PM, Junchao Zhang 
>>> wrote:
>>>>>> 
>>>>>> 
>>>>>> On Fri, Apr 20, 2018 at 3:18 PM, Matthew Knepley 
>>> wrote:
>>>>>> On Fri, Apr 20, 2018 at 4:10 PM, Junchao Zhang 
>>> wrote:
>>>>>> To pad a vector, i.e., copy a vector to a new one, I have to call
>>> VecSetValue(newb,1,&idx,...) for each element. But to be efficient, what I
>>> really needs is to set a block of values in one call. It looks PETSc does
>>> not have a routine for that(?). I looked at VecSetValuesBlocked, but it
>>> looks it is not for that purpose.
>>>>>> Should we have something like VecSetValuesBlock(Vec v,PetscInt
>>> i,PetscInt cnt,PetscScalar *value, InsertMode mode) to set cnt values
>>> starting at index i?
>>>>>> 
>>>>>> Use VecGetArray().
>>>>>> Did you mean VecGetArray b and newb, do a memcpy from b to new and
>>> then restore them? If yes, it does not work since some of the values I want
>>> to set might be remote.
>>>>>> E.g, I have 4 processors. b's size is 181 and is distributed as 46,
>>> 45,45,45, newb is distributed as 48,45,45,45 to match a matrix of block
>>> size 3.
>>>>>> 
>>>>>> 
>>>>>>  Matt
>>>>>> 
>>>>>> --Junchao Zhang
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> --
>>>>>> What most experimenters take for granted before they begin their
>>> experiments is infinitely more interesting than any results to which their
>>> experiments lead.
>>>>>> -- Norbert Wiener
>>>>>> 
>>>>>> https://www.cse.buffalo.edu/~knepley/
>>>>>> 
>>>>> 
>>>>> 
>>> 

[petsc-dev] broken stuff in next for many days

[Smith, Barry F.]

ftp://ftp.mcs.anl.gov/pub/petsc/nightlylogs/archive/2018/04/21/examples_next_arch-osx-xsdk-opt_ipro.log

not ok diff-dm_impls_plex_tutorials-ex5_2a_nsize-8_format-hdf5_xdmf
#   6,9c6,9
#   < [0]: 7 > 3
#   < [0]: 7 > 4
#   < [0]: 7 > 5
#   < [0]: 7 > 6
#   ---
#   > [0]: 7 > 0
#   > [0]: 7 > 1
#   > [0]: 8 > 1
#   > [0]: 8 > 2
#   11,13c11,16
#   < [0]: 8 > 6

   Is someone working on fixing this or does the branch need to be abandoned 
and next cleaned, this has been haunting next for way to long.

   Barry

Re: [petsc-dev] broken stuff in next for many days

[Smith, Barry F.]

> On Apr 21, 2018, at 6:38 PM, Balay, Satish  wrote:
> 
> On Sat, 21 Apr 2018, Václav Hapla wrote:
> 
>> Sorry Barry, that's my fault.
>> 
>> Thanks a lot Satish for the quick fix with !trilinos. Hope it works now.
>> 
>> I think the chaco partitioner should be definitely replaced by simple it 
>> these tests as Matt suggested. But I will rather push it within a separate 
>> PR since it extends the tested archs (by removing chaco and !trilinos 
>> requirements) so some previously unseen issues might appear theoretically.
> 
> Yes - any changes should be a new PR.
> 
> It would be good to figure out exactly where the difference comes form
> [wrt chaco from trilinos vs separate install of chaco - or some other
> interaction from trilinos]

   Different random numbers being generated?
> 
> and we could keep chaco tests - and tailor them appropriately with
> '!trilinos' or have multipe outout file [i.e _alt.out format] for
> chaco tests - if thats appropriate.

   If both sets are correct than an _alt file is better than !trilinos in the 
long run.

Barry


> 
> Satish

Re: [petsc-dev] PetscOptions defaults after PetscInitialize

[Smith, Barry F.]

  Yuck, I think a far better user API is that PetscOptionsInsertFile() be 
callable before PetscInitialize(). The problem is that people have shoveled so 
much post-PetscInitialize() code into  PetscOptionsInsertFile() over the years 
that stripping it all out would be painful. Maybe get a simplier pre- 2005 
version of the routine and strip out the post-PetscInitialize() material?

   Barry


> On Apr 22, 2018, at 5:54 PM, Jed Brown  wrote:
> 
> For users that read their own configuration files and/or choose
> PetscOptionsInsertFile after PetscInitialize, we don't have a good way
> to avoid overwriting PETSC_OPTIONS or command-line options.  The user
> could manually find argv and the environment variable, but that's a poor
> abstraction.  Should PetscOptionsInsertFile learn how to behave so as to
> add new entries to the options database, but not supersede any that
> already exist?

Re: [petsc-dev] PetscOptions defaults after PetscInitialize

[Smith, Barry F.]

> On Apr 22, 2018, at 7:41 PM, Jed Brown  wrote:
> 
> "Smith, Barry F."  writes:
> 
>>  Yuck, I think a far better user API is that PetscOptionsInsertFile() be 
>> callable before PetscInitialize(). The problem is that people have shoveled 
>> so much post-PetscInitialize() code into  PetscOptionsInsertFile() over the 
>> years that stripping it all out would be painful. Maybe get a simplier pre- 
>> 2005 version of the routine and strip out the post-PetscInitialize() 
>> material?
> 
> You want every rank to open the file independently?

   I forgot about that.

>  Or
> PetscOptionsInsertFile somehow caches the file contents without using
> PetscMalloc and broadcasts it after reaching PetscInitialize?  That
> seems a bit crazy.

   Maybe have use of pre-PetscInitialize() PetscOptionsInsertFile() cache the 
__file name(s)__
and then have PetscInitialize() loop over the cached filenames and load those 
options using simple MPI 
and PetscOptionsSetValue() before processing the command line etc
 
   Post-PetscInitialize() use of PetscOptionsInsertFile() would remain the same 
as today. We'd still need
a stripped down version of PetscOptionsInsertFile().

   Your proposed route allows bad decisions made years ago to dictate a bad 
user API forever. I don't like that.

  Barry

 
> 
>>   Barry
>> 
>> 
>>> On Apr 22, 2018, at 5:54 PM, Jed Brown  wrote:
>>> 
>>> For users that read their own configuration files and/or choose
>>> PetscOptionsInsertFile after PetscInitialize, we don't have a good way
>>> to avoid overwriting PETSC_OPTIONS or command-line options.  The user
>>> could manually find argv and the environment variable, but that's a poor
>>> abstraction.  Should PetscOptionsInsertFile learn how to behave so as to
>>> add new entries to the options database, but not supersede any that
>>> already exist?

Re: [petsc-dev] PetscOptions defaults after PetscInitialize

[Smith, Barry F.]

> On Apr 22, 2018, at 7:45 PM, Matthew Knepley  wrote:
> 
> On Sun, Apr 22, 2018 at 8:41 PM, Jed Brown  wrote:
> "Smith, Barry F."  writes:
> 
> >   Yuck, I think a far better user API is that PetscOptionsInsertFile() be 
> > callable before PetscInitialize(). The problem is that people have shoveled 
> > so much post-PetscInitialize() code into  PetscOptionsInsertFile() over the 
> > years that stripping it all out would be painful. Maybe get a simplier pre- 
> > 2005 version of the routine and strip out the post-PetscInitialize() 
> > material?
> 
> You want every rank to open the file independently?  Or
> PetscOptionsInsertFile somehow caches the file contents without using
> PetscMalloc and broadcasts it after reaching PetscInitialize?  That
> seems a bit crazy.
> 
> I am not for this "before PetcInitialize" strategy at all.

   Why not. It was just stupidity on my part 20 years ago that I didn't think 
to make the various set options functions callable before PetscInitialize(). 
Better just to fix that oversight.

> 
> However, I do think its far too fat. It initializes everything we can think 
> of without
> giving the user and entry points in to customize it. That is compiler-level 
> user disempowerment.
> 
> I would rather see us rearchitect Init() so that you have better control over 
> options, logging,
> CUDA, and all the other initializations hiding in there.

   Make a concrete proposal. "better control" is pretty vague.

> 
>Matt
>  
> >Barry
> >
> >
> >> On Apr 22, 2018, at 5:54 PM, Jed Brown  wrote:
> >> 
> >> For users that read their own configuration files and/or choose
> >> PetscOptionsInsertFile after PetscInitialize, we don't have a good way
> >> to avoid overwriting PETSC_OPTIONS or command-line options.  The user
> >> could manually find argv and the environment variable, but that's a poor
> >> abstraction.  Should PetscOptionsInsertFile learn how to behave so as to
> >> add new entries to the options database, but not supersede any that
> >> already exist?
> 
> 
> 
> -- 
> What most experimenters take for granted before they begin their experiments 
> is infinitely more interesting than any results to which their experiments 
> lead.
> -- Norbert Wiener
> 
> https://www.cse.buffalo.edu/~knepley/

Re: [petsc-dev] PETSc blame digest (next) 2018-04-23

[Smith, Barry F.]

   Dang, I fixed this yesterday but then forgot to merge the branch 
barry/bugfix-petscformatconvert into next for testing.

   Barry


> On Apr 23, 2018, at 7:28 AM, PETSc checkBuilds 
>  wrote:
> 
> 
> 
> Dear PETSc developer,
> 
> This email contains listings of contributions attributed to you by
> `git blame` that caused compiler errors or warnings in PETSc automated
> testing.  Follow the links to see the full log files. Please attempt to fix
> the issues promptly or let us know at petsc-dev@mcs.anl.gov if you are unable
> to resolve the issues.
> 
> Thanks,
>  The PETSc development team
> 
> 
> 
> warnings attributed to commit 
> https://bitbucket.org/petsc/petsc/commits/14416c0
> Insure PetscPrintf(), PetscVPrintfDefault(), PetscSynchronizedPrintf(), 
> PetscSynchronizedFPrintf(), and PetscViewerASCIIPrintf() handle printed 
> strings longer than the default buffer size of 8*1024 bytes properly
> 
>  src/sys/classes/viewer/impls/ascii/filev.c:991
>
> [http://ftp.mcs.anl.gov/pub/petsc/nightlylogs/archive/2018/04/23/build_next_arch-freebsd-cxx-cmplx-pkgs-dbg_wii.log]
>  
> /usr/home/balay/petsc.next/src/sys/classes/viewer/impls/ascii/filev.c:991:20: 
> warning: comparison between signed and unsigned integer expressions 
> [-Wsign-compare]
>
> [http://ftp.mcs.anl.gov/pub/petsc/nightlylogs/archive/2018/04/23/build_next_arch-linux-pkgs-cxx-mlib_el6.log]
>  
> /home/sandbox/petsc/petsc.next-3/src/sys/classes/viewer/impls/ascii/filev.c:991:43:
>  warning: comparison between signed and unsigned integer expressions 
> [-Wsign-compare]
>
> [http://ftp.mcs.anl.gov/pub/petsc/nightlylogs/archive/2018/04/23/build_next_arch-linux-cmplx-gcov_steamroller.log]
>  
> /sandbox/petsc/petsc.next-2/src/sys/classes/viewer/impls/ascii/filev.c:991:43:
>  warning: comparison between signed and unsigned integer expressions 
> [-Wsign-compare]
>
> [http://ftp.mcs.anl.gov/pub/petsc/nightlylogs/archive/2018/04/23/build_next_arch-linux-xsdk-dbg_cg.log]
>  
> /sandbox/petsc/petsc.next/src/sys/classes/viewer/impls/ascii/filev.c:991:43: 
> warning: comparison between signed and unsigned integer expressions 
> [-Wsign-compare]
>
> [http://ftp.mcs.anl.gov/pub/petsc/nightlylogs/archive/2018/04/23/build_next_arch-linux-opt-cxx-quad_grind.log]
>  
> /sandbox/petsc/petsc.next-3/src/sys/classes/viewer/impls/ascii/filev.c:991:43:
>  warning: comparison between signed and unsigned integer expressions 
> [-Wsign-compare]
>
> [http://ftp.mcs.anl.gov/pub/petsc/nightlylogs/archive/2018/04/23/build_next_arch-freebsd-cxx-pkgs-opt_wii.log]
>  
> /usr/home/balay/petsc.next-2/src/sys/classes/viewer/impls/ascii/filev.c:991:20:
>  warning: comparison between signed and unsigned integer expressions 
> [-Wsign-compare]
> 
>  src/sys/fileio/mprint.c:785
>
> [http://ftp.mcs.anl.gov/pub/petsc/nightlylogs/archive/2018/04/23/build_next_arch-linux-matlab-ilp64-gcov_thrash.log]
>  /sandbox/petsc/petsc.next/src/sys/fileio/mprint.c:785:17: error: 'len' 
> undeclared (first use in this function)
>
> [http://ftp.mcs.anl.gov/pub/petsc/nightlylogs/archive/2018/04/23/build_next_arch-linux-matlab-ilp64-gcov_thrash.log]
>  /sandbox/petsc/petsc.next/src/sys/fileio/mprint.c:785:12: warning: 
> unused variable 'buff' [-Wunused-variable]
> 
> 
> 
> warnings attributed to commit 
> https://bitbucket.org/petsc/petsc/commits/df41390
> fixed problem with Matlab being nonresponsive when run with -nodisplay and 
> built with MPI
> 
>  src/sys/fileio/mprint.c:788
>
> [http://ftp.mcs.anl.gov/pub/petsc/nightlylogs/archive/2018/04/23/build_next_arch-linux-matlab-ilp64-gcov_thrash.log]
>  /sandbox/petsc/petsc.next/src/sys/fileio/mprint.c:788:20: error: 'buf' 
> undeclared (first use in this function)
> 
> 
> To opt-out from receiving these messages - send a request to 
> petsc-dev@mcs.anl.gov.

Re: [petsc-dev] Chaco from --download-chaco vs --download-trilinos (Re: broken stuff in next for many days)

[Smith, Barry F.]

s in 
>> formatting to identify crucial changes easily. But definitely these two 
>> versions differ and can give different results.
>> 
>> The main intent of my tests in src/dm/impls/plex/examples/tutorials/ex5.c is 
>> testing HDF5 I/O with different formats and I just needed to somehow 
>> distribute the sequential mesh read from exodus file. So I will replace 
>> chaco by simple here. But still it's a question whether we want to do 
>> something about the confusing situation with Chaco.
>> 
>> Vaclav
>> 
>>> 22. 4. 2018 v 17:56, Satish Balay :
>>> 
>>> On Sun, 22 Apr 2018, Smith, Barry F. wrote:
>>> 
>>>>> It would be good to figure out exactly where the difference comes form
>>>>> [wrt chaco from trilinos vs separate install of chaco - or some other
>>>>> interaction from trilinos]
>>>> 
>>>>  Different random numbers being generated?
>>> 
>>> Its trilinos vs non-trilinos build of packages - so I don't think its 
>>> 'different random number generation' issue.
>>> 
>>> Trilinos install does:
>>> 
>>>   if self.libraries.check(self.dlib, "interface"):
>>> self.addDefine('HAVE_CHACO',1)
>>> self.addDefine('HAVE_CHACO_INT_ASSIGNMENT',1)
>>>   if self.libraries.check(self.dlib, "ML_Set_PrintLevel"):
>>> self.addDefine('HAVE_ML',1)
>>>   if self.libraries.check(self.dlib, "Zoltan_LB_Partition"):
>>> self.addDefine('HAVE_ZOLTAN',1)
>>>   if self.libraries.check(self.dlib, "ex_close"):
>>> self.addDefine('HAVE_EXODUSII',1)
>>> 
>>> 
>>> For one there is different chaco code in petsc based on this flag
>>> HAVE_CHACO_INT_ASSIGNMENT. Its not clear to me if this is causing the
>>> difference.
>>> 
>>> --download-chaco is using 'Chaco-2.2-p2.tar.gz'. This matches the
>>> latest chaco tarball from their website [Chaco-2.2.tar.gz]. So I do
>>> not know if chaco packaged by trilinos has changes that cause this
>>> difference [clearly it added some change that resulted in us using the
>>> flag HAVE_CHACO_INT_ASSIGNMENT]
>>> 
>>> And then there is also exodus.. --download-exodus uses the latest
>>> snapshot from https://github.com/gsjaardema/seacas/
>>> 
>>> Satish
>> 
>> 
> 

Re: [petsc-dev] Chaco from --download-chaco vs --download-trilinos (Re: broken stuff in next for many days)

[Smith, Barry F.]

   Let's just have the alt file for the chaco diffs and leave everything else 
the way it. I can't see the effort to change builds stuff worth it in this case.

   Barry


> On Apr 23, 2018, at 2:13 PM, Balay, Satish  wrote:
> 
> I was thinking more in terms of: our download options should match
> source distribution model of externalpackages [but not necessarily
> build all components]
> 
> I see chaco is primarily part of seacas and not distributed separately.
> [and then seacas is redistributed in trilinos]
> 
> So how about:
> 
> --download-seacas [build chaco and exodusii]
> 
> And remove the separate package support for exodusii and chaco [i.e
> remove --download-exodusii --download-chaco]
> 
> [and then somehow make the seacas build via trilinos match the standalone 
> build]
> 
> I see zoltan is distributed separately, in seacas and also in trilinos
> [so current --download-zoltan is still ok?]
> 
> ML is the odd one out - I guess we'll have to extract/repackage the
> latest version from trilinos - and use it with --download-ml.
> 
> Satish
> 
> On Mon, 23 Apr 2018, Smith, Barry F. wrote:
> 
>> 
>>  Making people install all of Trilinos for just Chaco or just ML is just 
>> plain mean and we are not mean.
>> 
>>   Suitesparse is at least not huge and is well contained so just because we 
>> install all of it doesn't mean we should require the same route for Trilinos.
>> 
>>   Barry
>> 
>> 
>> 
>>> On Apr 23, 2018, at 10:11 AM, Balay, Satish  wrote:
>>> 
>>> Ah - yes. I did check the difference in petsc code wrt
>>> PETSC_HAVE_CHACO_INT_ASSIGNMENT - but didn't check the details inside
>>> trilinos/chaco. So the available standalone chaco tarball is version
>>> 2.2 but the version included inside seacas is 3.0.0 [with a switch
>>> from short to int]
>>> 
>>> We have these hierarchy of packages - and need to figure out a better
>>> [install] interface so the same version is used irrespective of the
>>> mode of install?
>>> 
>>> Right now - --download-exodusii gets and installs seacas [with only
>>> exodusii enabled]
>>> 
>>> So we have:
>>> 
>>> --download-ml --download-chaco --download-zoltan [separate packages]
>>> 
>>> --download-exodusii [installs seacas - and enables only exodusii]
>>> 
>>> --download-trilinos [installs trilinos which includes 
>>> seacas(exodusii,chaco), ml - zoltan]
>>> 
>>> since seacas is distributed separately and includes exodusii and chaco - 
>>> have:
>>> 
>>> --download-seacas=1 [with options exodusii=1,chaco=1 by default [or 
>>> disabled?]
>>> 
>>> --download-seacas=1
>>> --download-seacas=1 --download-seacas-exodusii=0 [only install chaco]
>>> --download-seacas-chaco=0 [only install exodusii]
>>> 
>>> [but zoltan, ml are not distributed separately?]
>>> 
>>> Or should we always use trilinos? - if one wants any of the above
>>> packages [a huge download]? and enable or disable selected
>>> sub-components as desired?
>>> 
>>> Another similar package is SuiteSparse. At some point we had install
>>> for only sub-components(cholmod,umfpack) - and then merged them all
>>> into --download-suitesparse.
>>> 
>>> Satish
>>> 
>>> On Mon, 23 Apr 2018, Vaclav Hapla wrote:
>>> 
>>>> I can confirm --download-chaco vs --download-trilinos give different chaco 
>>>> partitioning - I can reproduce on my machine. So I made some investigation.
>>>> 
>>>> PETSC_HAVE_CHACO_INT_ASSIGNMENT is set only for --download-trilinos. 
>>>> This macro makes difference only at 4 places:
>>>> 
>>>> At src/mat/partition/impls/chaco/chaco.c:8 and 
>>>> src/dm/impls/plex/plexpartition.c:1192:
>>>> #if defined(PETSC_HAVE_CHACO_INT_ASSIGNMENT)
>>>> #include 
>>>> #else
>>>> /* Older versions of Chaco do not have an include file */
>>>> PETSC_EXTERN int interface(int nvtxs, int *start, int *adjacency, int 
>>>> *vwgts,
>>>>float *ewgts, float *x, float *y, float *z, char 
>>>> *outassignname,
>>>>char *outfilename, short *assignment, int architecture, 
>>>> int ndims_tot,
>>>>int mesh_dims[3], double *goal, int global_method, int 
>>>> local_method,
>>>>int rqi_flag, int vmax, int nd

Re: [petsc-dev] Install location for MPIUNI mpi.h

[Smith, Barry F.]

> On Apr 25, 2018, at 1:36 PM, Jed Brown  wrote:
> 
> It is currently installed to include/petsc/mpiuni/mpi.h and petscsys.h
> includes it as , which means that users of MPIUNI need to put
> -I/prefix/include/petsc/mpiuni in their command lines.  Matt and I agree
> that this is bad.  We disagree on the solution.
> 
> He wants to install it to /prefix/include/mpi.h as though the user had
> written --download-mpich.  This would conflict if a user later installs
> a real MPI to that location.

  Jed,

   So you propose in petscsys.h ?

#if defined(PETSC_HAVE_MPIUNI)
#include  
#else
#include 
#endif

  I don't have a problem with this.

I notice that the Fortran petscsys.h is scattered full of weird MPIUni specific 
stuff like

#if defined (PETSC_HAVE_MPIUNI)
#include "mpiunifdef.h"
#endif

#if defined(PETSC_HAVE_MPIUNI)
#define MPI_Comm PetscFortranInt
#define MPI_Group PetscFortranInt
#define PetscMPIInt PetscFortranInt
#else
#define MPI_Comm integer
#define MPI_Group integer
#define PetscMPIInt integer
#endif

It seems PETSc Fortran does not use the standard mpif.h file? 

   Barry




> 
> I would rather that petscsys.h include  because it
> can't be used without PETSc and nobody who ever wrote #include 
> in their own code will be happy if they get MPIUNI.

Re: [petsc-dev] Install location for MPIUNI mpi.h

[Smith, Barry F.]

> On Apr 25, 2018, at 2:31 PM, Jed Brown  wrote:
> 
> "Smith, Barry F."  writes:
> 
>>> On Apr 25, 2018, at 1:36 PM, Jed Brown  wrote:
>>> 
>>> It is currently installed to include/petsc/mpiuni/mpi.h and petscsys.h
>>> includes it as , which means that users of MPIUNI need to put
>>> -I/prefix/include/petsc/mpiuni in their command lines.  Matt and I agree
>>> that this is bad.  We disagree on the solution.
>>> 
>>> He wants to install it to /prefix/include/mpi.h as though the user had
>>> written --download-mpich.  This would conflict if a user later installs
>>> a real MPI to that location.
>> 
>>  Jed,
>> 
>>   So you propose in petscsys.h ?
>> 
>> #if defined(PETSC_HAVE_MPIUNI)
>> #include  
>> #else
>> #include 
>> #endif
>> 
>>  I don't have a problem with this.
> 
> Yes, and same installation layout as today.

   Ok, this is far better than copying the mpiuni mpi.h file to a public place 
(Matt's suggestion) and is a bit better than requiring the extra -I flag 
(Satish's suggestion) 

> 
>> I notice that the Fortran petscsys.h is scattered full of weird MPIUni 
>> specific stuff like
>> 
>> #if defined (PETSC_HAVE_MPIUNI)
>> #include "mpiunifdef.h"
>> #endif
>> 
>> #if defined(PETSC_HAVE_MPIUNI)
>> #define MPI_Comm PetscFortranInt
>> #define MPI_Group PetscFortranInt
>> #define PetscMPIInt PetscFortranInt
>> #else
>> #define MPI_Comm integer
>> #define MPI_Group integer
>> #define PetscMPIInt integer
>> #endif
>> 
>> It seems PETSc Fortran does not use the standard mpif.h file? 
> 
> Yuck.

   I don't remember the history of this construct. There must have been a 
reason years ago that may no longer exist, I don't know.

   Barry

Re: [petsc-dev] Install location for MPIUNI mpi.h

[Smith, Barry F.]

> On Apr 25, 2018, at 2:40 PM, Jed Brown  wrote:
> 
> "Smith, Barry F."  writes:
> 
>>> On Apr 25, 2018, at 2:31 PM, Jed Brown  wrote:
>>> 
>>> "Smith, Barry F."  writes:
>>> 
>>>>> On Apr 25, 2018, at 1:36 PM, Jed Brown  wrote:
>>>>> 
>>>>> It is currently installed to include/petsc/mpiuni/mpi.h and petscsys.h
>>>>> includes it as , which means that users of MPIUNI need to put
>>>>> -I/prefix/include/petsc/mpiuni in their command lines.  Matt and I agree
>>>>> that this is bad.  We disagree on the solution.
>>>>> 
>>>>> He wants to install it to /prefix/include/mpi.h as though the user had
>>>>> written --download-mpich.  This would conflict if a user later installs
>>>>> a real MPI to that location.
>>>> 
>>>> Jed,
>>>> 
>>>>  So you propose in petscsys.h ?
>>>> 
>>>> #if defined(PETSC_HAVE_MPIUNI)
>>>> #include  
>>>> #else
>>>> #include 
>>>> #endif
>>>> 
>>>> I don't have a problem with this.
>>> 
>>> Yes, and same installation layout as today.
>> 
>>   Ok, this is far better than copying the mpiuni mpi.h file to a
>>   public place (Matt's suggestion) and is a bit better than requiring
>>   the extra -I flag (Satish's suggestion)
> 
> Is this okay for 'maint'?  It feels aggressive, but I'm having trouble
> constructing a scenario in which it would break an existing installation
> or existing code.

Seems ok to me.

Re: [petsc-dev] Install location for MPIUNI mpi.h

[Smith, Barry F.]

> On Apr 25, 2018, at 2:47 PM, Balay, Satish  wrote:
> 
> On Wed, 25 Apr 2018, Smith, Barry F. wrote:
> 
>> 
>> 
>>> On Apr 25, 2018, at 2:31 PM, Jed Brown  wrote:
>>> 
>>> "Smith, Barry F."  writes:
>>> 
>>>>> On Apr 25, 2018, at 1:36 PM, Jed Brown  wrote:
>>>>> 
>>>>> It is currently installed to include/petsc/mpiuni/mpi.h and petscsys.h
>>>>> includes it as , which means that users of MPIUNI need to put
>>>>> -I/prefix/include/petsc/mpiuni in their command lines.  Matt and I agree
>>>>> that this is bad.  We disagree on the solution.
>>>>> 
>>>>> He wants to install it to /prefix/include/mpi.h as though the user had
>>>>> written --download-mpich.  This would conflict if a user later installs
>>>>> a real MPI to that location.
>>>> 
>>>> Jed,
>>>> 
>>>>  So you propose in petscsys.h ?
>>>> 
>>>> #if defined(PETSC_HAVE_MPIUNI)
>>>> #include  
>>>> #else
>>>> #include 
>>>> #endif
>>>> 
>>>> I don't have a problem with this.
>>> 
>>> Yes, and same installation layout as today.
>> 
>>   Ok, this is far better than copying the mpiuni mpi.h file to a public 
>> place (Matt's suggestion) and is a bit better than requiring the extra -I 
>> flag (Satish's suggestion) 
>> 
> 
> All approaches have drawbacks. This approach breaks user code. I guess thats 
> easy to work arround [and fix these
> examples awell]
> 
> $ git grep '' |grep examples
> src/dm/examples/tests/ex42.c:#include 
> src/tao/leastsquares/examples/tutorials/chwirut2.c:#include 

  Fix these examples at the same time.
> 
> 
>>> 
>>>> I notice that the Fortran petscsys.h is scattered full of weird MPIUni 
>>>> specific stuff like
>>>> 
>>>> #if defined (PETSC_HAVE_MPIUNI)
>>>> #include "mpiunifdef.h"
>>>> #endif
>>>> 
>>>> #if defined(PETSC_HAVE_MPIUNI)
>>>> #define MPI_Comm PetscFortranInt
>>>> #define MPI_Group PetscFortranInt
>>>> #define PetscMPIInt PetscFortranInt
>>>> #else
>>>> #define MPI_Comm integer
>>>> #define MPI_Group integer
>>>> #define PetscMPIInt integer
>>>> #endif
> 
> I might have diverged from mpi standard to prevent '-i8' affecting 
> petsc+mpiuni code.

   Ah, yes

> 
> Satish
> 
>>>> 
>>>> It seems PETSc Fortran does not use the standard mpif.h file? 
>>> 
>>> Yuck.
>> 
>>   I don't remember the history of this construct. There must have been a 
>> reason years ago that may no longer exist, I don't know.
>> 
>>   Barry

Re: [petsc-dev] [petsc-users] Segmentation Violation in getting DMPlex coordinates

[Smith, Barry F.]

> On Apr 29, 2018, at 6:19 AM, Patrick Sanan  wrote:
> 
> For functions like this (only for one impl), should this new check be 
> considered new best practices (as opposed to the composition approach, 
> defining things with names like DMDASetUniformCoordinates_DMDA())? It seems 
> like less boilerplate, as well as avoiding a function on the stack (and the 
> check itself if it's turned off in optimized mode).

No, it is just error checking for incorrectly written code that is too much 
hassle to change. 

   Barry

> 
> 2018-04-28 22:38 GMT+02:00 Smith, Barry F. :
> 
>   Added runtime error checking for such incorrect calls in 
> barry/dmda-calls-type-check
> 
> 
> > On Apr 28, 2018, at 9:19 AM, Matthew Knepley  wrote:
> > 
> > On Sat, Apr 28, 2018 at 2:08 AM, Danyang Su  wrote:
> > Hi All,
> > 
> > I use DMPlex and need to get coordinates back after distribution. However, 
> > I always get segmentation violation in getting coords values in the 
> > following codes if using multiple processors. If only one processor is 
> > used, it works fine.
> > 
> > For each processors, the off value starts from 0 which looks good. I also 
> > tried 0-based index, which gives the same error. Would any one help to 
> > check what is wrong here?
> > 
> >  idof   1 off   0
> >  idof   2 off   0
> >  idof   1 off   2
> >  idof   2 off   2
> >  idof   1 off   4
> >  idof   2 off   4
> >  idof   1 off   6
> >  idof   2 off   6
> >  idof   1 off   8
> >  idof   2 off   8
> > 
> > 
> >   DM :: distributedMesh, cda
> >   Vec :: gc
> >   PetscScalar, pointer :: coords(:)
> >   PetscSection ::  cs
> > 
> >   ...
> > 
> >   call DMGetCoordinatesLocal(dmda_flow%da,gc,ierr)
> >   CHKERRQ(ierr)
> > 
> >   call DMGetCoordinateDM(dmda_flow%da,cda,ierr)
> >   CHKERRQ(ierr)
> > 
> >   call DMGetDefaultSection(cda,cs,ierr)
> >   CHKERRQ(ierr)
> > 
> >   call PetscSectionGetChart(cs,istart,iend,ierr)
> >   CHKERRQ(ierr)
> > 
> >   !c get coordinates array
> >   call DMDAVecGetArrayF90(cda,gc,coords,ierr)
> > 
> > You cannot call DMDA function if you have a DMPlex. You jsut call 
> > VecGetArrayF90()
> > 
> >Matt
> >  
> >   CHKERRQ(ierr)
> > 
> >   do ipoint = istart, iend-1
> > 
> > call PetscSectionGetDof(cs,ipoint,dof,ierr)
> > CHKERRQ(ierr)
> > 
> > call PetscSectionGetOffset(cs,ipoint,off,ierr)
> > CHKERRQ(ierr)
> > 
> > inode = ipoint-istart+1
> > 
> > if (cell_coords == coords_xyz) then
> >   nodes(inode)%x = coords(off+1)
> >   nodes(inode)%y = coords(off+2)
> >   nodes(inode)%z = coords(off+3)
> > else if (cell_coords == coords_xy) then
> >   nodes(inode)%x = coords(off+1)
> >   nodes(inode)%y = coords(off+2)
> >   nodes(inode)%z = 0.0d0
> > else if (cell_coords == coords_yz) then
> >   nodes(inode)%x = 0.0d0
> >   nodes(inode)%y = coords(off+1)
> >   nodes(inode)%z = coords(off+2)
> > else if (cell_coords ==coords_xz) then
> >   nodes(inode)%x = coords(off+1)
> >   nodes(inode)%y = 0.0d0
> >   nodes(inode)%z = coords(off+2)
> > end if
> >   end do
> > 
> >   call DMDAVecRestoreArrayF90(cda,gc,coords,ierr)
> >   CHKERRQ(ierr)
> > 
> > Thanks,
> > 
> > Danyang
> > 
> > 
> > 
> > 
> > 
> > -- 
> > What most experimenters take for granted before they begin their 
> > experiments is infinitely more interesting than any results to which their 
> > experiments lead.
> > -- Norbert Wiener
> > 
> > https://www.cse.buffalo.edu/~knepley/
> 
> 

Re: [petsc-dev] Error in MatGetDiagonal_Shell?

[Smith, Barry F.]

   I don't understand

The code has

  ierr = MatGetDiagonal(A,W2);CHKERRQ(ierr);
  ierr = VecView(W2,viewer);CHKERRQ(ierr);

   the output from this VecView seems to match the diagonal of the matrix 
printed earlier.

> On Apr 29, 2018, at 10:02 AM, Stefano Zampini  
> wrote:
> 
> By looking at src/mat/examples/tests/output/ex88_1.out, it seems that the 
> output of MatGetDiagonal is wrong for MATSHELL. 
> 
> it should be 
> 2.96782e+08
> 1.41731e+09
> and not (as computed) 
> 1.10918e+08
> 2.06459e+08

   What do you mean "as computed"? The output file does not contain
> 1.10918e+08
> 2.06459e+08
anywhere.

   Are you modifying the example and running it again to get the incorrect 
value?

  Barry

> 
> -- 
> Stefano

Re: [petsc-dev] Error in MatGetDiagonal_Shell?

[Smith, Barry F.]

https://bitbucket.org/petsc/petsc/pull-requests/951/barry-fix-matgetdiagonal-shell/diff



> On Apr 30, 2018, at 12:38 AM, Stefano Zampini  
> wrote:
> 
> 
>> On Apr 30, 2018, at 3:14 AM, Smith, Barry F.  wrote:
>> 
>> 
>>  I don't understand
>> 
>>   The code has
>> 
>> ierr = MatGetDiagonal(A,W2);CHKERRQ(ierr);
>> ierr = VecView(W2,viewer);CHKERRQ(ierr);
>> 
>>  the output from this VecView seems to match the diagonal of the matrix 
>> printed earlier.
>> 
>>> On Apr 29, 2018, at 10:02 AM, Stefano Zampini  
>>> wrote:
>>> 
>>> By looking at src/mat/examples/tests/output/ex88_1.out, it seems that the 
>>> output of MatGetDiagonal is wrong for MATSHELL. 
>>> 
>>> it should be 
>>> 2.96782e+08
>>> 1.41731e+09
>>> and not (as computed) 
>>> 1.10918e+08
>>> 2.06459e+08
>> 
>>  What do you mean "as computed"? The output file does not contain
>>> 1.10918e+08
>>> 2.06459e+08
>> anywhere.
>> 
>>  Are you modifying the example and running it again to get the incorrect 
>> value?
> 
> Yes, my bad. I did not explain it correctly
> From maint, if you just run ex88 you will get the incorrect output, which is 
> not caught in the nightlyies because floating point values are completely 
> ignored.
> 
> [szampini@localhost tests]$ ./ex88
> Matrix of type: shell
> Mat Object: 1 MPI processes
>  type: seqdense
> 2.96781903e+08 3.92880081e+08 
> 6.13746835e+08 1.417314158000e+09 
> Vec Object: 1 MPI processes
>  type: seq
> 3.92973e+19
> 8.54197e+19
> Vec Object: 1 MPI processes
>  type: seq
> 1.10918e+08
> 2.06459e+08
> MatDiagonalSet not tested on MATSHELL
> ….
> 
>> 
>> Barry
>> 
>>> 
>>> -- 
>>> Stefano
>> 
> 

Re: [petsc-dev] GCC8 Fortran length changes from int to size_t

[Smith, Barry F.]

   Jeff, (and others),

 Do you know of a tool that can take a C prototype and automatically 
generate the Fortran C binding interface definition? We currently generate 
stubs for C functions that have character arguments manually and it would be 
great to remove that manual step.

   Thanks

  Barry


> On May 2, 2018, at 11:42 PM, Jeff Hammond  wrote:
> 
> Or you could just use ISO_C_BINDING.  Decent compilers should support it.
> 
> On Wed, May 2, 2018 at 8:56 AM, Jed Brown  wrote:
> See Fortran Language Issues.
> 
>   https://gcc.gnu.org/gcc-8/porting_to.html
> 
> We'll have to test for this (probably compiler version) and change the
> PETSC_MIXED_LEN / PETSC_END_LEN to use size_t instead of int.
> 
> 
> 
> -- 
> Jeff Hammond
> jeff.scie...@gmail.com
> http://jeffhammond.github.io/

Re: [petsc-dev] variable 'length' is uninitialized in src/sys/fileio/mprint.c

[Smith, Barry F.]

   Thanks. I have removed the function in commit 08486c302b2  since it is dead, 
unneeded code that I should have removed before.

   Barry


> On May 3, 2018, at 10:19 AM, Vaclav Hapla  wrote:
> 
> Barry,
> this is obviously wrong in current master:
> 
> 14416c0e507 src/sys/fileio/mprint.c 791) size_t length;
> cb500232d0b src/sys/fileio/mprint.c 792) char   buff[length];
> 
> It results in
>  warning: variable 'length' is uninitialized when used here [-Wuninitialized]
> when the PETSC_HAVE_MATLAB_ENGINE macro is defined.
> 
> Vaclav

Re: [petsc-dev] GCC8 Fortran length changes from int to size_t

[Smith, Barry F.]

> On May 7, 2018, at 3:56 AM, Lisandro Dalcin  wrote:
> 
> On Mon, 7 May 2018 at 03:11, Jed Brown  wrote:
> 
>> Lisandro Dalcin  writes:
> 
> How do you exactly want to implement that? Totally replace these
> special functions with the BIND(C) interface that calls directly the
> C
> function, or rather generate a native Fortran subroutine that calls
> the C function through a BIND(C) interface?
>>> 
 The latter.  Usage should not change for existing Fortran users.
>>> 
>>> Things may end up being involved anyway, even if using ISO_C_BINDINGS.
>>> 
>>> http://fortranwiki.org/fortran/show/Generating+C+Interfaces#strings
> 
>> I guess I don't understand.  Are you concerned about the string copying?
>> Note that FIXCHAR (used in current interfaces) allocates memory.
> 
> Well, no, my point is that an automatic interface generator based on
> ISO_C_BINDINGS (as Barry wanted) may prove slightly harder to implement
> than expected.

   I see no benefit in using the ISO_C_BINDINGS for the Fortran functions with 
char * arguments either automatically or manually so let's stick to the current 
model.

   Barry

> 
> 
> -- 
> Lisandro Dalcin
> 
> Research Scientist
> Computer, Electrical and Mathematical Sciences & Engineering (CEMSE)
> Extreme Computing Research Center (ECRC)
> King Abdullah University of Science and Technology (KAUST)
> http://ecrc.kaust.edu.sa/
> 
> 4700 King Abdullah University of Science and Technology
> al-Khawarizmi Bldg (Bldg 1), Office # 0109
> Thuwal 23955-6900, Kingdom of Saudi Arabia
> http://www.kaust.edu.sa
> 
> Office Phone: +966 12 808-0459

Re: [petsc-dev] calling petsc from a mex file

[Smith, Barry F.]

> On May 8, 2018, at 12:30 PM, Munson, Todd  wrote:
> 
> 
> Hi,
> 
> I wonder if anyone has experience calling petsc from matlab using mex.  I 
> have the 
> simplest possible mexFunction that just calls PetscInitialize() and 
> PetscFinalize().  That part seems to work fine, but when I exit 
> from matlab I get a segmentation violation and a petsc error
> message.

   What are all the error messages? Cut and paste. I've done this in the past 
but not for years.

> 
> I have tried compiling both with --download-mpich and --with-mpi=0 with the
> same outcome.
> 
> Maybe I am missing something flags I need to set when configuring petsc?
> 
> Any help would be appreciated.
> 
> My backup plan is to rewrite the matlab code that wants to invoke a petsc 
> solver so 
> that I can make it work using a PetscMatlabEngine instead.
> 
> Thanks, Todd.
> 

Re: [petsc-dev] calling petsc from a mex file

[Smith, Barry F.]

  Push the branch that does this or send me the code and I'll debug it. I now 
have some vague memory about this behavior.

Barry


> On May 8, 2018, at 1:53 PM, Munson, Todd  wrote:
> 
> 
> 
>> On May 8, 2018, at 1:02 PM, Smith, Barry F.  wrote:
>> 
>> 
>> 
>>> On May 8, 2018, at 12:30 PM, Munson, Todd  wrote:
>>> 
>>> 
>>> Hi,
>>> 
>>> I wonder if anyone has experience calling petsc from matlab using mex.  I 
>>> have the 
>>> simplest possible mexFunction that just calls PetscInitialize() and 
>>> PetscFinalize().  That part seems to work fine, but when I exit 
>>> from matlab I get a segmentation violation and a petsc error
>>> message.
>> 
>>  What are all the error messages? Cut and paste. I've done this in the past 
>> but not for years.
> 
> Below is the matlab session and error message.  I do note that if I issue 
> PetscFinalize() and 
> that command completes, the Petsc signal handlers remain installed/active.  
> If I do a
> PetscPopSignalHandler() immediately after PetscInitialize(), I do not get the 
> error
> message, but a "Segmentation fault: 11" output at the end.
> 
> Because petsc is being loaded as a dynamic library, are there possibly 
> uninitialized global
> variables or state?
> 
> Thanks, Todd.
> 
> [0]PETSC ERROR: 
> 
> [0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, 
> probably memory access out of range
> [0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
> [0]PETSC ERROR: or see 
> http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind
> [0]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to 
> find memory corruption errors
> [0]PETSC ERROR: likely location of problem given in stack below
> [0]PETSC ERROR: -  Stack Frames 
> 
> [0]PETSC ERROR: Note: The EXACT line numbers in the stack are not available,
> [0]PETSC ERROR:   INSTEAD the line number of the start of the function
> [0]PETSC ERROR:   is given.
> [0]PETSC ERROR: - Error Message 
> --
> [0]PETSC ERROR: Signal received
> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for 
> trouble shooting.
> [0]PETSC ERROR: Petsc Development GIT revision: v3.9.1-409-gf553c6556e  GIT 
> Date: 2018-05-07 17:17:17 -0500
> [0]PETSC ERROR: Unknown Name on a arch-darwin-c-debug named 
> eduroam062-089.wl.anl-external.org by tmunson Tue May  8 12:04:45 2018
> [0]PETSC ERROR: Configure options --download-mpich CC=/usr/bin/clang 
> CXX=/usr/bin/clang++ FC=/opt/local/bin/gfortran
> [0]PETSC ERROR: #1 User provided function() line 0 in  unknown file
> application called MPI_Abort(MPI_COMM_WORLD, 59) - process 0
> [unset]: write_line error; fd=-1 buf=:cmd=abort exitcode=59
> :
> system msg for write_line failure : Bad file descriptor

Re: [petsc-dev] calling petsc from a mex file

[Smith, Barry F.]

  Huh

$ matlab -nodisplay

  < M A T L A B (R) >
Copyright 1984-2018 The MathWorks, Inc.
 R2018a (9.4.0.813654) 64-bit (maci64)
   February 23, 2018

Warning: Name is nonexistent or not a directory: 
/Users/barrysmith/Src/nodal-dg/Codes1.1 
Warning: Name is nonexistent or not a directory: 
/Users/barrysmith/Src/nodal-dg/Codes1.1/CFD2D 
Warning: Name is nonexistent or not a directory: 
/Users/barrysmith/Src/nodal-dg/Codes1.1 
 
To get started, type one of these: helpwin, helpdesk, or demo.
For product information, visit www.mathworks.com.
 
>> taopounders()
>> exit
~/Src/petsc/src/tao/leastsquares/examples/matlab 
(tmunson/tao-pounders-matlab-interface=) arch-matlab


> On May 8, 2018, at 2:30 PM, Munson, Todd  wrote:
> 
> 
> Its in tmunson/tao-pounders-matlab-interface
> 
> Most of the code is commented out or otherwise inactive.  There is a hacked
> command in src/tao/leastsquares/examples/matlab that I use to compile the
> mex file.
> 
> To test, you should be able to start matlab and say "taopounders(); exit".
> 
> Todd.
> 
>> On May 8, 2018, at 2:21 PM, Smith, Barry F.  wrote:
>> 
>> 
>> Push the branch that does this or send me the code and I'll debug it. I now 
>> have some vague memory about this behavior.
>> 
>>   Barry
>> 
>> 
>>> On May 8, 2018, at 1:53 PM, Munson, Todd  wrote:
>>> 
>>> 
>>> 
>>>> On May 8, 2018, at 1:02 PM, Smith, Barry F.  wrote:
>>>> 
>>>> 
>>>> 
>>>>> On May 8, 2018, at 12:30 PM, Munson, Todd  wrote:
>>>>> 
>>>>> 
>>>>> Hi,
>>>>> 
>>>>> I wonder if anyone has experience calling petsc from matlab using mex.  I 
>>>>> have the 
>>>>> simplest possible mexFunction that just calls PetscInitialize() and 
>>>>> PetscFinalize().  That part seems to work fine, but when I exit 
>>>>> from matlab I get a segmentation violation and a petsc error
>>>>> message.
>>>> 
>>>> What are all the error messages? Cut and paste. I've done this in the past 
>>>> but not for years.
>>> 
>>> Below is the matlab session and error message.  I do note that if I issue 
>>> PetscFinalize() and 
>>> that command completes, the Petsc signal handlers remain installed/active.  
>>> If I do a
>>> PetscPopSignalHandler() immediately after PetscInitialize(), I do not get 
>>> the error
>>> message, but a "Segmentation fault: 11" output at the end.
>>> 
>>> Because petsc is being loaded as a dynamic library, are there possibly 
>>> uninitialized global
>>> variables or state?
>>> 
>>> Thanks, Todd.
>>> 
>>> [0]PETSC ERROR: 
>>> 
>>> [0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, 
>>> probably memory access out of range
>>> [0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
>>> [0]PETSC ERROR: or see 
>>> http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind
>>> [0]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X 
>>> to find memory corruption errors
>>> [0]PETSC ERROR: likely location of problem given in stack below
>>> [0]PETSC ERROR: -  Stack Frames 
>>> 
>>> [0]PETSC ERROR: Note: The EXACT line numbers in the stack are not available,
>>> [0]PETSC ERROR:   INSTEAD the line number of the start of the function
>>> [0]PETSC ERROR:   is given.
>>> [0]PETSC ERROR: - Error Message 
>>> --
>>> [0]PETSC ERROR: Signal received
>>> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for 
>>> trouble shooting.
>>> [0]PETSC ERROR: Petsc Development GIT revision: v3.9.1-409-gf553c6556e  GIT 
>>> Date: 2018-05-07 17:17:17 -0500
>>> [0]PETSC ERROR: Unknown Name on a arch-darwin-c-debug named 
>>> eduroam062-089.wl.anl-external.org by tmunson Tue May  8 12:04:45 2018
>>> [0]PETSC ERROR: Configure options --download-mpich CC=/usr/bin/clang 
>>> CXX=/usr/bin/clang++ FC=/opt/local/bin/gfortran
>>> [0]PETSC ERROR: #1 User provided function() line 0 in  unknown file
>>> application called MPI_Abort(MPI_COMM_WORLD, 59) - process 0
>>> [unset]: write_line error; fd=-1 buf=:cmd=abort exitcode=59
>>> :
>>> system msg for write_line failure : Bad file descriptor
>> 
> 

Re: [petsc-dev] Proposed changes to TS API

[Smith, Barry F.]

> On May 10, 2018, at 4:12 PM, Jed Brown  wrote:
> 
> "Zhang, Hong"  writes:
> 
>> Dear PETSc folks,
>> 
>> Current TS APIs (IFunction/IJacobian+RHSFunction/RHSJacobian) were designed 
>> for the fully implicit formulation F(t,U,Udot) = G(t,U).
>> Shampine's paper 
>> (https://www.sciencedirect.com/science/article/pii/S0377042706004110?via%3Dihub)
>>  explains some reasoning behind it.
>> 
>> Our formulation is general enough to cover all the following common cases
>> 
>>  *   Udot = G(t,U) (classic ODE)
>>  *   M Udot = G(t,U)  (ODEs/DAEs for mechanical and electronic systems)
>>  *   M(t,U) Udot = G(t,U) (PDEs)
>> 
>> Yet the TS APIs provide the capability to solve both simple problems and 
>> complicated problems. However, we are not doing well to make TS easy to use 
>> and efficient especially for simple problems. Over the years, we have 
>> clearly seen the main drawbacks including:
>> 1. The shift parameter exposed in IJacobian is terribly confusing, 
>> especially to new users. Also it is not conceptually straightforward when 
>> using AD or finite differences on IFunction to approximate IJacobian.
> 
> What isn't straightforward about AD or FD on the IFunction?  That one
> bit of chain rule?
> 
>> 2. It is difficult to switch from fully implicit to fully explicit. Users 
>> cannot use explicit methods when they provide IFunction/IJacobian.
> 
> This is a real issue, but it's extremely common for PDE to have boundary
> conditions enforced as algebraic constraints, thus yielding a DAE.
> 
>> 3. The structure of mass matrix is completely invisible to TS. This means 
>> giving up all the opportunities to improve efficiency. For example, when M 
>> is constant or weekly dependent on U, we might not want to evaluate/update 
>> it every time IJacobian is called. If M is diagonal, the Jacobian can be 
>> shifted more efficiently than just using MatAXPY().
> 
> I don't understand 
> 
>> 4. Reshifting the Jacobian is unnecessarily expensive and sometimes buggy.
> 
> Why is "reshifting" needed?  After a step is rejected and when the step
> size changes for a linear constant operator?
> 
>> Consider the scenario below.
>> shift = a;
>>  TSComputeIJacobian()
>>  shift = b;
>>  TSComputeIJacobian() // with the same U and Udot as last call
>> Changing the shift parameter requires the Jacobian function to be evaluated 
>> again. If users provide only RHSJacobian, the Jacobian will not be 
>> updated/reshifted in the second call because TSComputeRHSJacobian() finds 
>> out that U has not been changed. This issue is fixable by adding more logic 
>> into the already convoluted implementation of TSComputeIJacobian(), but the 
>> intention here is to illustrate the cumbersomeness of current IJacobian and 
>> the growing complications in TSComputeIJacobian() that IJacobian causes.
>> 
>> So I propose that we have two separate matrices dF/dUdot and dF/dU, and 
>> remove the shift parameter from IJacobian. dF/dU will be calculated by 
>> IJacobian; dF/dUdot will be calculated by a new callback function and 
>> default to an identity matrix if it is not provided by users. Then the users 
>> do not need to assemble the shifted Jacobian since TS will handle the 
>> shifting whenever needed. And knowing the structure of dF/dUdot (the mass 
>> matrix), TS will become more flexible. In particular, we will have
>> 
>>  *   easy access to the unshifted Jacobian dF/dU (this simplifies the 
>> adjoint implementation a lot),
> 
> How does this simplify the adjoint?
> 
>>  *   plenty of opportunities to optimize TS when the mass matrix is diagonal 
>> or constant or weekly dependent on U (which accounts for almost all use 
>> cases in practice),
> 
> But longer critical path,

   What do you mean by longer critical path?

> more storage required, and more data motion.

  The extra storage needed is related to the size of the mass matrix correct? 
And the extra data motion is related to the size of the mass matrix correct?

   Is the extra work coming from a needed call to MatAXPY (to combine the 
scaled mass matrix with the Jacobian) in Hong's approach? While, in theory, the 
user can avoid the MatAXPY in the current code if they have custom code that 
assembles directly the scaled mass matrix and Jacobian? But surely most users 
would not write such custom code and would themselves keep a copy of the mass 
matrix (likely constant) and use MatAXPY() to combine the copy of the mass 
matrix with the Jacobian they compute at each timestep/stage?  Or am I missing 
something?

> And if the mass matrix is simple, won't it take a very small fraction of
> time, thus have little gains from "optimizing it"?

   Is the Function approach only theoretically much more efficient than Hong's 
approach when the mass matrix is nontrivial? That is the mass matrix has a 
nonzero structure similar to the Jacobian? 
> 
>>  *   easy conversion from fully implicit to fully expl

Re: [petsc-dev] Proposed changes to TS API

[Smith, Barry F.]

> On May 11, 2018, at 7:05 AM, Matthew Knepley  wrote:
> 
> On Fri, May 11, 2018 at 12:25 AM, Smith, Barry F.  wrote:
> 
> 
> > On May 10, 2018, at 4:12 PM, Jed Brown  wrote:
> > 
> > "Zhang, Hong"  writes:
> > 
> >> Dear PETSc folks,
> >> 
> >> Current TS APIs (IFunction/IJacobian+RHSFunction/RHSJacobian) were 
> >> designed for the fully implicit formulation F(t,U,Udot) = G(t,U).
> >> Shampine's paper 
> >> (https://www.sciencedirect.com/science/article/pii/S0377042706004110?via%3Dihub<https://www.sciencedirect.com/science/article/pii/S0377042706004110?via=ihub>)
> >>  explains some reasoning behind it.
> >> 
> >> Our formulation is general enough to cover all the following common cases
> >> 
> >>  *   Udot = G(t,U) (classic ODE)
> >>  *   M Udot = G(t,U)  (ODEs/DAEs for mechanical and electronic systems)
> >>  *   M(t,U) Udot = G(t,U) (PDEs)
> >> 
> >> Yet the TS APIs provide the capability to solve both simple problems and 
> >> complicated problems. However, we are not doing well to make TS easy to 
> >> use and efficient especially for simple problems. Over the years, we have 
> >> clearly seen the main drawbacks including:
> >> 1. The shift parameter exposed in IJacobian is terribly confusing, 
> >> especially to new users. Also it is not conceptually straightforward when 
> >> using AD or finite differences on IFunction to approximate IJacobian.
> > 
> > What isn't straightforward about AD or FD on the IFunction?  That one
> > bit of chain rule?
> > 
> >> 2. It is difficult to switch from fully implicit to fully explicit. Users 
> >> cannot use explicit methods when they provide IFunction/IJacobian.
> > 
> > This is a real issue, but it's extremely common for PDE to have boundary
> > conditions enforced as algebraic constraints, thus yielding a DAE.
> > 
> >> 3. The structure of mass matrix is completely invisible to TS. This means 
> >> giving up all the opportunities to improve efficiency. For example, when M 
> >> is constant or weekly dependent on U, we might not want to evaluate/update 
> >> it every time IJacobian is called. If M is diagonal, the Jacobian can be 
> >> shifted more efficiently than just using MatAXPY().
> > 
> > I don't understand 
> > 
> >> 4. Reshifting the Jacobian is unnecessarily expensive and sometimes buggy.
> > 
> > Why is "reshifting" needed?  After a step is rejected and when the step
> > size changes for a linear constant operator?
> > 
> >> Consider the scenario below.
> >> shift = a;
> >>  TSComputeIJacobian()
> >>  shift = b;
> >>  TSComputeIJacobian() // with the same U and Udot as last call
> >> Changing the shift parameter requires the Jacobian function to be 
> >> evaluated again. If users provide only RHSJacobian, the Jacobian will not 
> >> be updated/reshifted in the second call because TSComputeRHSJacobian() 
> >> finds out that U has not been changed. This issue is fixable by adding 
> >> more logic into the already convoluted implementation of 
> >> TSComputeIJacobian(), but the intention here is to illustrate the 
> >> cumbersomeness of current IJacobian and the growing complications in 
> >> TSComputeIJacobian() that IJacobian causes.
> >> 
> >> So I propose that we have two separate matrices dF/dUdot and dF/dU, and 
> >> remove the shift parameter from IJacobian. dF/dU will be calculated by 
> >> IJacobian; dF/dUdot will be calculated by a new callback function and 
> >> default to an identity matrix if it is not provided by users. Then the 
> >> users do not need to assemble the shifted Jacobian since TS will handle 
> >> the shifting whenever needed. And knowing the structure of dF/dUdot (the 
> >> mass matrix), TS will become more flexible. In particular, we will have
> >> 
> >>  *   easy access to the unshifted Jacobian dF/dU (this simplifies the 
> >> adjoint implementation a lot),
> > 
> > How does this simplify the adjoint?
> > 
> >>  *   plenty of opportunities to optimize TS when the mass matrix is 
> >> diagonal or constant or weekly dependent on U (which accounts for almost 
> >> all use cases in practice),
> > 
> > But longer critical path,
> 
>What do you mean by longer critical path?
> 
> > more storage required, and more data motion.
> 
>   The extra storage needed is related to the size of the mass matrix cor

Re: [petsc-dev] Proposed changes to TS API

[Smith, Barry F.]

> On May 11, 2018, at 8:03 AM, Stefano Zampini  
> wrote:
> 
> I don’t think changing the current TS API is best approach.
> 
> Obtaining separate Jacobians is a need for adjoints and tangent linear models 
> only.
> This is how I implemented it in stefano_zampini/feature-continuousadjoint
> https://bitbucket.org/petsc/petsc/src/7203629c61cbeced536ed8ba1dd2ef85ffb89e8f/src/ts/interface/tssplitjac.c#lines-48
> 
> Note that, instead of requiring the user to call PetscObjectComposeFunction, 
> we can use a function pointer and have TSSetComputeSplitJacobians

   So you are proposing keeping TSSetIFunction and TSSetIJacobian and ADDING a 
new API TSSetComputeSplitJacobians() and it that is not provided calling 
TSComputeIJacobian() twice with different shifts (which is definitely not 
efficient and is what Hong does also).

   Barry

> 
> 
> 
>> On May 11, 2018, at 3:20 PM, Jed Brown  wrote:
>> 
>> "Smith, Barry F."  writes:
>> 
>>>> On May 10, 2018, at 4:12 PM, Jed Brown  wrote:
>>>> 
>>>> "Zhang, Hong"  writes:
>>>> 
>>>>> Dear PETSc folks,
>>>>> 
>>>>> Current TS APIs (IFunction/IJacobian+RHSFunction/RHSJacobian) were 
>>>>> designed for the fully implicit formulation F(t,U,Udot) = G(t,U).
>>>>> Shampine's paper 
>>>>> (https://www.sciencedirect.com/science/article/pii/S0377042706004110?via%3Dihub<https://www.sciencedirect.com/science/article/pii/S0377042706004110?via=ihub>)
>>>>>  explains some reasoning behind it.
>>>>> 
>>>>> Our formulation is general enough to cover all the following common cases
>>>>> 
>>>>> *   Udot = G(t,U) (classic ODE)
>>>>> *   M Udot = G(t,U)  (ODEs/DAEs for mechanical and electronic systems)
>>>>> *   M(t,U) Udot = G(t,U) (PDEs)
>>>>> 
>>>>> Yet the TS APIs provide the capability to solve both simple problems and 
>>>>> complicated problems. However, we are not doing well to make TS easy to 
>>>>> use and efficient especially for simple problems. Over the years, we have 
>>>>> clearly seen the main drawbacks including:
>>>>> 1. The shift parameter exposed in IJacobian is terribly confusing, 
>>>>> especially to new users. Also it is not conceptually straightforward when 
>>>>> using AD or finite differences on IFunction to approximate IJacobian.
>>>> 
>>>> What isn't straightforward about AD or FD on the IFunction?  That one
>>>> bit of chain rule?
>>>> 
>>>>> 2. It is difficult to switch from fully implicit to fully explicit. Users 
>>>>> cannot use explicit methods when they provide IFunction/IJacobian.
>>>> 
>>>> This is a real issue, but it's extremely common for PDE to have boundary
>>>> conditions enforced as algebraic constraints, thus yielding a DAE.
>>>> 
>>>>> 3. The structure of mass matrix is completely invisible to TS. This means 
>>>>> giving up all the opportunities to improve efficiency. For example, when 
>>>>> M is constant or weekly dependent on U, we might not want to 
>>>>> evaluate/update it every time IJacobian is called. If M is diagonal, the 
>>>>> Jacobian can be shifted more efficiently than just using MatAXPY().
>>>> 
>>>> I don't understand 
>>>> 
>>>>> 4. Reshifting the Jacobian is unnecessarily expensive and sometimes buggy.
>>>> 
>>>> Why is "reshifting" needed?  After a step is rejected and when the step
>>>> size changes for a linear constant operator?
>>>> 
>>>>> Consider the scenario below.
>>>>> shift = a;
>>>>> TSComputeIJacobian()
>>>>> shift = b;
>>>>> TSComputeIJacobian() // with the same U and Udot as last call
>>>>> Changing the shift parameter requires the Jacobian function to be 
>>>>> evaluated again. If users provide only RHSJacobian, the Jacobian will not 
>>>>> be updated/reshifted in the second call because TSComputeRHSJacobian() 
>>>>> finds out that U has not been changed. This issue is fixable by adding 
>>>>> more logic into the already convoluted implementation of 
>>>>> TSComputeIJacobian(), but the intention here is to illustrate the 
>>>>> cumbersomeness of current IJacobian and the growing complications in 
>>>>> TSComputeIJacobian() that IJacobian c

Re: [petsc-dev] Proposed changes to TS API

[Smith, Barry F.]

> On May 11, 2018, at 7:20 AM, Jed Brown  wrote:
> 
> "Smith, Barry F."  writes:
> 
>>> On May 10, 2018, at 4:12 PM, Jed Brown  wrote:
>>> 
>>> "Zhang, Hong"  writes:
>>> 
>>>> Dear PETSc folks,
>>>> 
>>>> Current TS APIs (IFunction/IJacobian+RHSFunction/RHSJacobian) were 
>>>> designed for the fully implicit formulation F(t,U,Udot) = G(t,U).
>>>> Shampine's paper 
>>>> (https://www.sciencedirect.com/science/article/pii/S0377042706004110?via%3Dihub<https://www.sciencedirect.com/science/article/pii/S0377042706004110?via=ihub>)
>>>>  explains some reasoning behind it.
>>>> 
>>>> Our formulation is general enough to cover all the following common cases
>>>> 
>>>> *   Udot = G(t,U) (classic ODE)
>>>> *   M Udot = G(t,U)  (ODEs/DAEs for mechanical and electronic systems)
>>>> *   M(t,U) Udot = G(t,U) (PDEs)
>>>> 
>>>> Yet the TS APIs provide the capability to solve both simple problems and 
>>>> complicated problems. However, we are not doing well to make TS easy to 
>>>> use and efficient especially for simple problems. Over the years, we have 
>>>> clearly seen the main drawbacks including:
>>>> 1. The shift parameter exposed in IJacobian is terribly confusing, 
>>>> especially to new users. Also it is not conceptually straightforward when 
>>>> using AD or finite differences on IFunction to approximate IJacobian.
>>> 
>>> What isn't straightforward about AD or FD on the IFunction?  That one
>>> bit of chain rule?
>>> 
>>>> 2. It is difficult to switch from fully implicit to fully explicit. Users 
>>>> cannot use explicit methods when they provide IFunction/IJacobian.
>>> 
>>> This is a real issue, but it's extremely common for PDE to have boundary
>>> conditions enforced as algebraic constraints, thus yielding a DAE.
>>> 
>>>> 3. The structure of mass matrix is completely invisible to TS. This means 
>>>> giving up all the opportunities to improve efficiency. For example, when M 
>>>> is constant or weekly dependent on U, we might not want to evaluate/update 
>>>> it every time IJacobian is called. If M is diagonal, the Jacobian can be 
>>>> shifted more efficiently than just using MatAXPY().
>>> 
>>> I don't understand 
>>> 
>>>> 4. Reshifting the Jacobian is unnecessarily expensive and sometimes buggy.
>>> 
>>> Why is "reshifting" needed?  After a step is rejected and when the step
>>> size changes for a linear constant operator?
>>> 
>>>> Consider the scenario below.
>>>> shift = a;
>>>> TSComputeIJacobian()
>>>> shift = b;
>>>> TSComputeIJacobian() // with the same U and Udot as last call
>>>> Changing the shift parameter requires the Jacobian function to be 
>>>> evaluated again. If users provide only RHSJacobian, the Jacobian will not 
>>>> be updated/reshifted in the second call because TSComputeRHSJacobian() 
>>>> finds out that U has not been changed. This issue is fixable by adding 
>>>> more logic into the already convoluted implementation of 
>>>> TSComputeIJacobian(), but the intention here is to illustrate the 
>>>> cumbersomeness of current IJacobian and the growing complications in 
>>>> TSComputeIJacobian() that IJacobian causes.
>>>> 
>>>> So I propose that we have two separate matrices dF/dUdot and dF/dU, and 
>>>> remove the shift parameter from IJacobian. dF/dU will be calculated by 
>>>> IJacobian; dF/dUdot will be calculated by a new callback function and 
>>>> default to an identity matrix if it is not provided by users. Then the 
>>>> users do not need to assemble the shifted Jacobian since TS will handle 
>>>> the shifting whenever needed. And knowing the structure of dF/dUdot (the 
>>>> mass matrix), TS will become more flexible. In particular, we will have
>>>> 
>>>> *   easy access to the unshifted Jacobian dF/dU (this simplifies the 
>>>> adjoint implementation a lot),
>>> 
>>> How does this simplify the adjoint?
>>> 
>>>> *   plenty of opportunities to optimize TS when the mass matrix is 
>>>> diagonal or constant or weekly dependent on U (which accounts for almost 
>>>> all use cases in practice),
>>> 
>>> But longer critical path,
>

Re: [petsc-dev] Proposed changes to TS API

[Smith, Barry F.]

> On May 11, 2018, at 10:17 AM, Smith, Barry F.  wrote:
> 
> 
> 
>> On May 11, 2018, at 7:05 AM, Matthew Knepley  wrote:
>> 
>> On Fri, May 11, 2018 at 12:25 AM, Smith, Barry F.  wrote:
>> 
>> 
>>> On May 10, 2018, at 4:12 PM, Jed Brown  wrote:
>>> 
>>> "Zhang, Hong"  writes:
>>> 
>>>> Dear PETSc folks,
>>>> 
>>>> Current TS APIs (IFunction/IJacobian+RHSFunction/RHSJacobian) were 
>>>> designed for the fully implicit formulation F(t,U,Udot) = G(t,U).
>>>> Shampine's paper 
>>>> (https://www.sciencedirect.com/science/article/pii/S0377042706004110?via%3Dihub<https://www.sciencedirect.com/science/article/pii/S0377042706004110?via=ihub>)
>>>>  explains some reasoning behind it.
>>>> 
>>>> Our formulation is general enough to cover all the following common cases
>>>> 
>>>> *   Udot = G(t,U) (classic ODE)
>>>> *   M Udot = G(t,U)  (ODEs/DAEs for mechanical and electronic systems)
>>>> *   M(t,U) Udot = G(t,U) (PDEs)
>>>> 
>>>> Yet the TS APIs provide the capability to solve both simple problems and 
>>>> complicated problems. However, we are not doing well to make TS easy to 
>>>> use and efficient especially for simple problems. Over the years, we have 
>>>> clearly seen the main drawbacks including:
>>>> 1. The shift parameter exposed in IJacobian is terribly confusing, 
>>>> especially to new users. Also it is not conceptually straightforward when 
>>>> using AD or finite differences on IFunction to approximate IJacobian.
>>> 
>>> What isn't straightforward about AD or FD on the IFunction?  That one
>>> bit of chain rule?
>>> 
>>>> 2. It is difficult to switch from fully implicit to fully explicit. Users 
>>>> cannot use explicit methods when they provide IFunction/IJacobian.
>>> 
>>> This is a real issue, but it's extremely common for PDE to have boundary
>>> conditions enforced as algebraic constraints, thus yielding a DAE.
>>> 
>>>> 3. The structure of mass matrix is completely invisible to TS. This means 
>>>> giving up all the opportunities to improve efficiency. For example, when M 
>>>> is constant or weekly dependent on U, we might not want to evaluate/update 
>>>> it every time IJacobian is called. If M is diagonal, the Jacobian can be 
>>>> shifted more efficiently than just using MatAXPY().
>>> 
>>> I don't understand 
>>> 
>>>> 4. Reshifting the Jacobian is unnecessarily expensive and sometimes buggy.
>>> 
>>> Why is "reshifting" needed?  After a step is rejected and when the step
>>> size changes for a linear constant operator?
>>> 
>>>> Consider the scenario below.
>>>> shift = a;
>>>> TSComputeIJacobian()
>>>> shift = b;
>>>> TSComputeIJacobian() // with the same U and Udot as last call
>>>> Changing the shift parameter requires the Jacobian function to be 
>>>> evaluated again. If users provide only RHSJacobian, the Jacobian will not 
>>>> be updated/reshifted in the second call because TSComputeRHSJacobian() 
>>>> finds out that U has not been changed. This issue is fixable by adding 
>>>> more logic into the already convoluted implementation of 
>>>> TSComputeIJacobian(), but the intention here is to illustrate the 
>>>> cumbersomeness of current IJacobian and the growing complications in 
>>>> TSComputeIJacobian() that IJacobian causes.
>>>> 
>>>> So I propose that we have two separate matrices dF/dUdot and dF/dU, and 
>>>> remove the shift parameter from IJacobian. dF/dU will be calculated by 
>>>> IJacobian; dF/dUdot will be calculated by a new callback function and 
>>>> default to an identity matrix if it is not provided by users. Then the 
>>>> users do not need to assemble the shifted Jacobian since TS will handle 
>>>> the shifting whenever needed. And knowing the structure of dF/dUdot (the 
>>>> mass matrix), TS will become more flexible. In particular, we will have
>>>> 
>>>> *   easy access to the unshifted Jacobian dF/dU (this simplifies the 
>>>> adjoint implementation a lot),
>>> 
>>> How does this simplify the adjoint?
>>> 
>>>> *   plenty of opportunities to optimize TS when the mass matrix is 
>>>> diagonal or constant or weekly dependen

Re: [petsc-dev] Proposed changes to TS API

[Smith, Barry F.]

> On May 11, 2018, at 12:58 PM, Stefano Zampini  
> wrote:
> 
> 
>> On May 11, 2018, at 6:20 PM, Smith, Barry F.  wrote:
>> 
>> 
>> 
>>> On May 11, 2018, at 8:03 AM, Stefano Zampini  
>>> wrote:
>>> 
>>> I don’t think changing the current TS API is best approach.
>>> 
>>> Obtaining separate Jacobians is a need for adjoints and tangent linear 
>>> models only.
>>> This is how I implemented it in stefano_zampini/feature-continuousadjoint
>>> https://bitbucket.org/petsc/petsc/src/7203629c61cbeced536ed8ba1dd2ef85ffb89e8f/src/ts/interface/tssplitjac.c#lines-48
>>> 
>>> Note that, instead of requiring the user to call 
>>> PetscObjectComposeFunction, we can use a function pointer and have 
>>> TSSetComputeSplitJacobians
>> 
>>  So you are proposing keeping TSSetIFunction and TSSetIJacobian and ADDING a 
>> new API TSSetComputeSplitJacobians() and it that is not provided calling 
>> TSComputeIJacobian() twice with different shifts (which is definitely not 
>> efficient and is what Hong does also).
>> 
> 
> Why is it inefficient?

   I meant calling the user ComputeIJacobian twice (to get the two parts) is 
inefficient. Not your API

> If you need BOTH dF/dUdot and dF/dU, you need two different assemblies even 
> if we change the API. Note that the physics is needed to evaluate dF/du, but 
> usually it’s not for dF/dUdot.
> And the MatAXPY is with SAME_NONZERO_PATTERN, so, basically no time.
> The only difference is one extra physics evaluation (that can be expensive). 
> However, advanced users that are aware of that, can provide their specialized 
> version of ComputeJacobians.
> 
> 
>>  Barry
>> 
>>> 
>>> 
>>> 
>>>> On May 11, 2018, at 3:20 PM, Jed Brown  wrote:
>>>> 
>>>> "Smith, Barry F."  writes:
>>>> 
>>>>>> On May 10, 2018, at 4:12 PM, Jed Brown  wrote:
>>>>>> 
>>>>>> "Zhang, Hong"  writes:
>>>>>> 
>>>>>>> Dear PETSc folks,
>>>>>>> 
>>>>>>> Current TS APIs (IFunction/IJacobian+RHSFunction/RHSJacobian) were 
>>>>>>> designed for the fully implicit formulation F(t,U,Udot) = G(t,U).
>>>>>>> Shampine's paper 
>>>>>>> (https://www.sciencedirect.com/science/article/pii/S0377042706004110?via%3Dihub<https://www.sciencedirect.com/science/article/pii/S0377042706004110?via=ihub>)
>>>>>>>  explains some reasoning behind it.
>>>>>>> 
>>>>>>> Our formulation is general enough to cover all the following common 
>>>>>>> cases
>>>>>>> 
>>>>>>> *   Udot = G(t,U) (classic ODE)
>>>>>>> *   M Udot = G(t,U)  (ODEs/DAEs for mechanical and electronic systems)
>>>>>>> *   M(t,U) Udot = G(t,U) (PDEs)
>>>>>>> 
>>>>>>> Yet the TS APIs provide the capability to solve both simple problems 
>>>>>>> and complicated problems. However, we are not doing well to make TS 
>>>>>>> easy to use and efficient especially for simple problems. Over the 
>>>>>>> years, we have clearly seen the main drawbacks including:
>>>>>>> 1. The shift parameter exposed in IJacobian is terribly confusing, 
>>>>>>> especially to new users. Also it is not conceptually straightforward 
>>>>>>> when using AD or finite differences on IFunction to approximate 
>>>>>>> IJacobian.
>>>>>> 
>>>>>> What isn't straightforward about AD or FD on the IFunction?  That one
>>>>>> bit of chain rule?
>>>>>> 
>>>>>>> 2. It is difficult to switch from fully implicit to fully explicit. 
>>>>>>> Users cannot use explicit methods when they provide IFunction/IJacobian.
>>>>>> 
>>>>>> This is a real issue, but it's extremely common for PDE to have boundary
>>>>>> conditions enforced as algebraic constraints, thus yielding a DAE.
>>>>>> 
>>>>>>> 3. The structure of mass matrix is completely invisible to TS. This 
>>>>>>> means giving up all the opportunities to improve efficiency. For 
>>>>>>> example, when M is constant or weekly dependent on U, we might not want 
>>>>>>> to evaluate/update it every time IJacobian is called. If M i

Re: [petsc-dev] Proposed changes to TS API

[Smith, Barry F.]

> On May 11, 2018, at 1:01 PM, Lisandro Dalcin  wrote:
> 
> On Fri, 11 May 2018 at 19:34, Jed Brown  wrote:
> 
>> "Smith, Barry F."  writes:
> 
>>>>> 
>>>>> I assemble the combined system directly.
>>>> 
>>>>  How, two sets of calls to MatSetValues(), One for the scaled mass
> matrix and one for the Jacobian entries? For a constant mass matrix does
> this mean you are recomputing the mass matrix entries each call? Or are you
> storing the mass matrix entries somewhere?  Or is your mass matrix diagonal
> only?
>>> 
>>>  Or do you build element by element the M*shift + J element stiffness
> and then insert it with a single MatSetValues() call?
> 
>> It isn't even built separately at the element scale, just summed
>> contributions at quadrature points.
> 
> That's exactly the way I do it in PetIGA, and the way I would do it in any
> other general-purpose FEM-like code. In high-order FEM, Jacobian assembly
> may very well account from 50% to 80% of runtime (at least that's my
> experience with PetIGA). IMHO, forcing users to have to do TWO global
> matrix assemblies per time step is simply unacceptable, both in terms of
> memory and runtime.
> 
> TS used to be an unusable pile of crap until Jed proposed the marvelous
> IJacobian interface. Suddenly COMPLEX fully-implicit DAE problems become
> SIMPLE to express, and a single IFunction/IJacobian pair reusable for
> different timestepper implementations a reality. And we got an added
> bounus: this was efficient, it involved a SINGLE global matrix assembly. If
> the issue is in supporting simpler problems, then we should go for an
> alternative interface with broken Jacobians, just as Stefano propossed in a
> previous email. I'm totally in favor of an additional broken Jacobians API,
> and totally againt the removal of the single-matrix IJacobian interface.

   Ok, thanks.

> 
> I don't buy the argument that IJacobian with shift is ugly,

   Not ugly, hard for most users to understand.

> and that such
> API drives users away from TS. At best, this is a documentation problem.
> Come on, this is just chain rule, should be kindergarden-level stuff for
> PETSc users. If we simplify things for the sake of making things simple for
> newcomers and beginners and make them annoyingly slow for power users
> solving complex problems, we will do a very bad business. This is not
> politically correct, but I'm much worried about loosing power users, you
> know, those that can eventually make a meaningful contributions to science
> and software projects. Beginners and newcomers eventually learn and benefit
> for common-sense software design, and will eventually appreciate it. I
> really hope populism to not win this battle :-)
> 
> -- 
> Lisandro Dalcin
> 
> Research Scientist
> Computer, Electrical and Mathematical Sciences & Engineering (CEMSE)
> Extreme Computing Research Center (ECRC)
> King Abdullah University of Science and Technology (KAUST)
> http://ecrc.kaust.edu.sa/
> 
> 4700 King Abdullah University of Science and Technology
> al-Khawarizmi Bldg (Bldg 1), Office # 0109
> Thuwal 23955-6900, Kingdom of Saudi Arabia
> http://www.kaust.edu.sa
> 
> Office Phone: +966 12 808-0459

Re: [petsc-dev] Proposed changes to TS API

[Smith, Barry F.]

> On May 11, 2018, at 3:36 PM, Jed Brown  wrote:
> 
> "Zhang, Hong"  writes:
> 
>> We are not forcing users to do two matrix assemblies per time
>> step. For most cases, there is even no need to update dF/dUdot at
>> all. For extreme cases that the application requires frequent update
>> on dF/dUdot and assembly is expensive, one can always assemble the
>> single-matrix Jacobian and throw it to SNES directly.
> 
> For the vast majority of cases, the mass matrix terms are inexpensive
> and can be summed during each assembly at negligible cost (cheaper than
> accessing those terms from an already-assembled matrix).
> 
>> 
>> TS used to be an unusable pile of crap until Jed proposed the marvelous
>> IJacobian interface. Suddenly COMPLEX fully-implicit DAE problems become
>> SIMPLE to express, and a single IFunction/IJacobian pair reusable for
>> different timestepper implementations a reality. And we got an added
>> bounus: this was efficient, it involved a SINGLE global matrix assembly. If
>> the issue is in supporting simpler problems, then we should go for an
>> alternative interface with broken Jacobians, just as Stefano propossed in a
>> previous email. I'm totally in favor of an additional broken Jacobians API,
>> and totally againt the removal of the single-matrix IJacobian interface.
>> 
>> The current IJacobian is essentially SNESJacobian. And the single-matrix 
>> SNESJacobian interface is always there. Power users could set up the 
>> SNESJacobian directly if we pass a read-only shift parameter to them. So we 
>> are by no means prohibiting power users from doing what they want.
> 
> You mean the user would call TSGetSNES and SNESSetJacobian, then
> internally call TSGetIJacobianShift and some new function to create
> Udot?  That sounds way messier and error-prone.
> 
>> IJacobian with shift mixes TS Jacobian and SNES Jacobian. This is the issue 
>> we need to fix.
> 
> It is just one interface producing exactly the matrix that the solver
> needs.

The implicit solver needs it, but the explicit solvers do not, nor do the 
adjoint computations. This is the issue.

Re: [petsc-dev] Proposed changes to TS API

[Smith, Barry F.]

> On May 11, 2018, at 3:38 PM, Matthew Knepley  wrote:
> 
> On Fri, May 11, 2018 at 4:36 PM, Jed Brown  wrote:
> "Zhang, Hong"  writes:
> 
> > We are not forcing users to do two matrix assemblies per time
> > step. For most cases, there is even no need to update dF/dUdot at
> > all. For extreme cases that the application requires frequent update
> > on dF/dUdot and assembly is expensive, one can always assemble the
> > single-matrix Jacobian and throw it to SNES directly.
> 
> For the vast majority of cases, the mass matrix terms are inexpensive
> and can be summed during each assembly at negligible cost (cheaper than
> accessing those terms from an already-assembled matrix).
> 
> >
> > TS used to be an unusable pile of crap until Jed proposed the marvelous
> > IJacobian interface. Suddenly COMPLEX fully-implicit DAE problems become
> > SIMPLE to express, and a single IFunction/IJacobian pair reusable for
> > different timestepper implementations a reality. And we got an added
> > bounus: this was efficient, it involved a SINGLE global matrix assembly. If
> > the issue is in supporting simpler problems, then we should go for an
> > alternative interface with broken Jacobians, just as Stefano propossed in a
> > previous email. I'm totally in favor of an additional broken Jacobians API,
> > and totally againt the removal of the single-matrix IJacobian interface.
> >
> > The current IJacobian is essentially SNESJacobian. And the single-matrix 
> > SNESJacobian interface is always there. Power users could set up the 
> > SNESJacobian directly if we pass a read-only shift parameter to them. So we 
> > are by no means prohibiting power users from doing what they want.
> 
> You mean the user would call TSGetSNES and SNESSetJacobian, then
> internally call TSGetIJacobianShift and some new function to create
> Udot?  That sounds way messier and error-prone.
> 
> And completely impossible to compose. We should be explicitly talking about 
> subsystems that use TS. For example,
> I have a nonlinear system for plasticity. I want to use a preconditioner that 
> introduces some elasticity and timesteps to
> steady state to provide a good Newton direction. I need to to be able to 
> create the solver without all sorts of digging
> in and setting things. This idea that you "just set SNESJacobian" is a 
> non-starter.

   But this is exactly what TSComputeIJacobian currently does, so is the 
current interface a non-starter?

> 
>   Matt
>  
> 
> > IJacobian with shift mixes TS Jacobian and SNES Jacobian. This is the issue 
> > we need to fix.
> 
> It is just one interface producing exactly the matrix that the solver
> needs.
> 
> 
> 
> -- 
> What most experimenters take for granted before they begin their experiments 
> is infinitely more interesting than any results to which their experiments 
> lead.
> -- Norbert Wiener
> 
> https://www.cse.buffalo.edu/~knepley/

Re: [petsc-dev] Proposed changes to TS API

[Smith, Barry F.]

> On May 11, 2018, at 5:09 PM, Jed Brown  wrote:
> 
> "Smith, Barry F."  writes:
> 
>>>> The current IJacobian is essentially SNESJacobian. And the single-matrix 
>>>> SNESJacobian interface is always there. Power users could set up the 
>>>> SNESJacobian directly if we pass a read-only shift parameter to them. So 
>>>> we are by no means prohibiting power users from doing what they want.
>>> 
>>> You mean the user would call TSGetSNES and SNESSetJacobian, then
>>> internally call TSGetIJacobianShift and some new function to create
>>> Udot?  That sounds way messier and error-prone.
>>> 
>>> And completely impossible to compose. We should be explicitly talking about 
>>> subsystems that use TS. For example,
>>> I have a nonlinear system for plasticity. I want to use a preconditioner 
>>> that introduces some elasticity and timesteps to
>>> steady state to provide a good Newton direction. I need to to be able to 
>>> create the solver without all sorts of digging
>>> in and setting things. This idea that you "just set SNESJacobian" is a 
>>> non-starter.
>> 
>>   But this is exactly what TSComputeIJacobian currently does, so is the 
>> current interface a non-starter?
> 
> It isn't at all the current interface.  

   You misunderstand what I am saying. The current interface produces exactly 
the Jacobian needed for the SNES solve (nothing more and nothing less) as a 
single monolithic matrix. Matt says this is a non-starter.

   Barry

> If the user is calling
> SNESSetJacobian, then we would need to open up the bowels of every
> SNESTSFormJacobian_* and make stable public APIs out of those internals
> (including the wiring for nonlinear multigrid).  This sounds like a
> terrible thing to make public.

Re: [petsc-dev] Proposed changes to TS API

[Smith, Barry F.]

> On May 11, 2018, at 5:26 PM, Jed Brown  wrote:
> 
> "Smith, Barry F."  writes:
> 
>>> On May 11, 2018, at 5:09 PM, Jed Brown  wrote:
>>> 
>>> "Smith, Barry F."  writes:
>>> 
>>>>>> The current IJacobian is essentially SNESJacobian. And the single-matrix 
>>>>>> SNESJacobian interface is always there. Power users could set up the 
>>>>>> SNESJacobian directly if we pass a read-only shift parameter to them. So 
>>>>>> we are by no means prohibiting power users from doing what they want.
>>>>> 
>>>>> You mean the user would call TSGetSNES and SNESSetJacobian, then
>>>>> internally call TSGetIJacobianShift and some new function to create
>>>>> Udot?  That sounds way messier and error-prone.
>>>>> 
>>>>> And completely impossible to compose. We should be explicitly talking 
>>>>> about subsystems that use TS. For example,
>>>>> I have a nonlinear system for plasticity. I want to use a preconditioner 
>>>>> that introduces some elasticity and timesteps to
>>>>> steady state to provide a good Newton direction. I need to to be able to 
>>>>> create the solver without all sorts of digging
>>>>> in and setting things. This idea that you "just set SNESJacobian" is a 
>>>>> non-starter.
>>>> 
>>>>  But this is exactly what TSComputeIJacobian currently does, so is the 
>>>> current interface a non-starter?
>>> 
>>> It isn't at all the current interface.  
>> 
>>   You misunderstand what I am saying. The current interface produces exactly 
>> the Jacobian needed for the SNES solve (nothing more and nothing less) as a 
>> single monolithic matrix. Matt says this is a non-starter.
> 
> Matt says that having the user call TSGetSNES and SNESSetJacobian (my
> interpretation of Hong's "users could set up the SNESJacobian directly")
> to assemble that matrix is a non-starter.

   He said it in a very confusing way and brought in seemly irrelevant 
discussion about some pseudo time stepping inside the needed nonlinear solve.

Re: [petsc-dev] Proposed changes to TS API

[Smith, Barry F.]

   Here is MY summary of the discussion so far.

1) the IFunction/IJacobian interface has its supporters. There is valid 
argument that for certain cases it can be more efficient than the proposed 
alternative; but this seems largely a theoretical believe at this time since 
there are no comparisons between nontrivial (or even trivial) codes that use 
the assembly directly the mass shift plus Jacobian vs the assembly separately 
and MatAXPY the two parts together.  As far as I can see this potential 
performance is the ONLY benefit to keeping the IFunction/IJacobian() interface? 
Please list additional benefits if there are any?

2) The IFunction/IJacobian approach makes it impossible for TS to access the 
mass matrix alone. 

3) But one can access the IJacobian matrix alone by passing a shift of zero

4) TSComputeIJacobian() is an ugly piece of shit code that has a confusing name 
since it also can incorporates the RHS Jacobian. 

4a) the TSComputeIJacobian() has issues for linear problems because it 
overwrites the user provided Jacobian and has hacks to deal with it

5) There is no standard way to solve M u = F() explicitly from the IFunction() 
form, (and cannot even with expressed with the explicit RHS approach) the user 
must manage the M solve themselves inside their RHSFunction.

6) There is some support for adding two new function callbacks that a) compute 
the mass matrix and b) compute the Jacobian part of the IJacobian. This appears 
particularly useful for implementing adjoints.

7) If we added the two new interfaces the internals of an already overly 
complicated TS become even more convoluted and unmaintainable.  For kicks I 
list the current TSComputeIJacobian() which I consider unmaintainable already.

  if (ijacobian) {
PetscBool missing;
PetscStackPush("TS user implicit Jacobian");
ierr = (*ijacobian)(ts,t,U,Udot,shift,A,B,ctx);CHKERRQ(ierr);
PetscStackPop;
ierr = MatMissingDiagonal(A,&missing,NULL);CHKERRQ(ierr);
if (missing) SETERRQ(PETSC_COMM_SELF,PETSC_ERR_ARG_WRONGSTATE,"Amat passed 
to TSSetIJacobian() must have all diagonal entries set, if they are zero you 
must still set them with a zero value");
if (B != A) {
  ierr = MatMissingDiagonal(B,&missing,NULL);CHKERRQ(ierr);
  if (missing) SETERRQ(PETSC_COMM_SELF,PETSC_ERR_ARG_WRONGSTATE,"Bmat 
passed to TSSetIJacobian() must have all diagonal entries set, if they are zero 
you must still set them with a zero value");
}
  }
  if (imex) {
if (!ijacobian) {  /* system was written as Udot = G(t,U) */
  PetscBool assembled;
  ierr = MatZeroEntries(A);CHKERRQ(ierr);
  ierr = MatAssembled(A,&assembled);CHKERRQ(ierr);
  if (!assembled) {
ierr = MatAssemblyBegin(A,MAT_FINAL_ASSEMBLY);CHKERRQ(ierr);
ierr = MatAssemblyEnd(A,MAT_FINAL_ASSEMBLY);CHKERRQ(ierr);
  }
  ierr = MatShift(A,shift);CHKERRQ(ierr);
  if (A != B) {
ierr = MatZeroEntries(B);CHKERRQ(ierr);
ierr = MatAssembled(B,&assembled);CHKERRQ(ierr);
if (!assembled) {
  ierr = MatAssemblyBegin(B,MAT_FINAL_ASSEMBLY);CHKERRQ(ierr);
  ierr = MatAssemblyEnd(B,MAT_FINAL_ASSEMBLY);CHKERRQ(ierr);
}
ierr = MatShift(B,shift);CHKERRQ(ierr);
  }
}
  } else {
Mat Arhs = NULL,Brhs = NULL;
if (rhsjacobian) {
  ierr = TSGetRHSMats_Private(ts,&Arhs,&Brhs);CHKERRQ(ierr);
  ierr = TSComputeRHSJacobian(ts,t,U,Arhs,Brhs);CHKERRQ(ierr);
}
if (Arhs == A) {   /* No IJacobian, so we only have the RHS matrix 
*/
  PetscBool flg;
  ts->rhsjacobian.scale = -1;
  ts->rhsjacobian.shift = shift;
  ierr = SNESGetUseMatrixFree(ts->snes,NULL,&flg);CHKERRQ(ierr);
  /* since -snes_mf_operator uses the full SNES function it does not need 
to be shifted or scaled here */
  if (!flg) {
ierr = MatScale(A,-1);CHKERRQ(ierr);
ierr = MatShift(A,shift);CHKERRQ(ierr);
  }
  if (A != B) {
ierr = MatScale(B,-1);CHKERRQ(ierr);
ierr = MatShift(B,shift);CHKERRQ(ierr);
  }
} else if (Arhs) {  /* Both IJacobian and RHSJacobian */
  MatStructure axpy = DIFFERENT_NONZERO_PATTERN;
  if (!ijacobian) { /* No IJacobian provided, but we have a 
separate RHS matrix */
ierr = MatZeroEntries(A);CHKERRQ(ierr);
ierr = MatShift(A,shift);CHKERRQ(ierr);
if (A != B) {
  ierr = MatZeroEntries(B);CHKERRQ(ierr);
  ierr = MatShift(B,shift);CHKERRQ(ierr);
}
  }
  ierr = MatAXPY(A,-1,Arhs,axpy);CHKERRQ(ierr);
  if (A != B) {
ierr = MatAXPY(B,-1,Brhs,axpy);CHKERRQ(ierr);
  }
}
  }

  Please comment and continue discussion.


> On May 11, 2018, at 5:09 PM, Jed Brown  wrote:
> 
> "Smith, Barry F."  writes:
> 
>>>> The current IJacobian is essentially SNESJacobian. And the single-matrix 
>>>> SNESJa

Re: [petsc-dev] Proposed changes to TS API

[Smith, Barry F.]

> On May 11, 2018, at 6:05 PM, Jed Brown  wrote:
> 
> "Smith, Barry F."  writes:
> 
>>   Here is MY summary of the discussion so far.
>> 
>> 1) the IFunction/IJacobian interface has its supporters. There is valid 
>> argument that for certain cases it can be more efficient than the proposed 
>> alternative; but this seems largely a theoretical believe at this time since 
>> there are no comparisons between nontrivial (or even trivial) codes that use 
>> the assembly directly the mass shift plus Jacobian vs the assembly 
>> separately and MatAXPY the two parts together.  As far as I can see this 
>> potential performance is the ONLY benefit to keeping the 
>> IFunction/IJacobian() interface? Please list additional benefits if there 
>> are any?
> 
> PCShell

   Please explain what this means, I have no clue.

> 
>> 2) The IFunction/IJacobian approach makes it impossible for TS to access the 
>> mass matrix alone. 
> 
> Without wasted work, yes.

  The computation of two Jacobians correct?
> 
>> 3) But one can access the IJacobian matrix alone by passing a shift of zero
>> 
>> 4) TSComputeIJacobian() is an ugly piece of shit code that has a confusing 
>> name since it also can incorporates the RHS Jacobian. 
> 
> If you get rid of that, then every implicit integrator will need to
> handle the RHSFunction/RHSJacobian itself.  It will be significantly
> more code to maintain.

   I don't propose "getting rid of it", I suggest perhaps the code could be 
refactored (I don't know how) to have something more streamlined.

> 
>> 4a) the TSComputeIJacobian() has issues for linear problems because it 
>> overwrites the user provided Jacobian and has hacks to deal with it
> 
> Yes, however that choice reduces memory usage which is sometimes an
> important factor.
> 
>> 5) There is no standard way to solve M u = F() explicitly from the 
>> IFunction() form, (and cannot even with expressed with the explicit RHS 
>> approach) the user must manage the M solve themselves inside their 
>> RHSFunction.
> 
> We could write this as an IMEX method with IFunction providing M u and
> RHSFunction providing F.  I think this could be specialized for constant
> M and provided automatically for any explicit method.
> 
>> 6) There is some support for adding two new function callbacks that a) 
>> compute the mass matrix and b) compute the Jacobian part of the IJacobian. 
>> This appears particularly useful for implementing adjoints.
>> 
>> 7) If we added the two new interfaces the internals of an already
>> overly complicated TS become even more convoluted and unmaintainable.  
> 
> We could split the shift into ashift and bshift (elided as always 1.0
> now), then users could opt to skip work if one of those was nonzero.
> That would be a modest generalization from what we have now and would
> enable any desired optimization.  Integrators that wish to take
> advantage of M not changing could interpret a flag saying that it was
> constant and then always call the IJacobian with ashift=0.  That would
> be unintrusive for existing users and still enable all optimizations.
> It's only one additional parameter and then optimized user code could
> look like
> 
>  for (each element) {
>Ke[m][m] = {};
>if (ashift != 0) {
>  Ke += ashift * (mass stuff);
>}
>if (bshift != 0) {
>  Ke += bshift * (non-mass stuff);
>}
>  }
> 
> Naive user code would elide the conditionals, but would still work with
> all integrators.

   Then also provide new TS developer routines such as TSComputeMass(), 
TSComputeJacobianWithoutMass() (bad name) so that TS code would look cleaner 
and not have a bunch of ugly calls to TSComputeIJacobian with shifts of 1 and 0 
around that I suspect Hong doesn't like.

Re: [petsc-dev] Proposed changes to TS API

[Smith, Barry F.]

> On May 11, 2018, at 6:05 PM, Jed Brown  wrote:
> 
> "Smith, Barry F."  writes:
> 
>>   Here is MY summary of the discussion so far.
>> 
>> 1) the IFunction/IJacobian interface has its supporters. There is valid 
>> argument that for certain cases it can be more efficient than the proposed 
>> alternative; but this seems largely a theoretical believe at this time since 
>> there are no comparisons between nontrivial (or even trivial) codes that use 
>> the assembly directly the mass shift plus Jacobian vs the assembly 
>> separately and MatAXPY the two parts together.  As far as I can see this 
>> potential performance is the ONLY benefit to keeping the 
>> IFunction/IJacobian() interface? Please list additional benefits if there 
>> are any?
> 
> PCShell
> 
>> 2) The IFunction/IJacobian approach makes it impossible for TS to access the 
>> mass matrix alone. 
> 
> Without wasted work, yes.
> 
>> 3) But one can access the IJacobian matrix alone by passing a shift of zero
>> 
>> 4) TSComputeIJacobian() is an ugly piece of shit code that has a confusing 
>> name since it also can incorporates the RHS Jacobian. 
> 
> If you get rid of that, then every implicit integrator will need to
> handle the RHSFunction/RHSJacobian itself.  It will be significantly
> more code to maintain.
> 
>> 4a) the TSComputeIJacobian() has issues for linear problems because it 
>> overwrites the user provided Jacobian and has hacks to deal with it
> 
> Yes, however that choice reduces memory usage which is sometimes an
> important factor.
> 
>> 5) There is no standard way to solve M u = F() explicitly from the 
>> IFunction() form, (and cannot even with expressed with the explicit RHS 
>> approach) the user must manage the M solve themselves inside their 
>> RHSFunction.
> 
> We could write this as an IMEX method with IFunction providing M u and
> RHSFunction providing F.  I think this could be specialized for constant
> M and provided automatically for any explicit method.
> 
>> 6) There is some support for adding two new function callbacks that a) 
>> compute the mass matrix and b) compute the Jacobian part of the IJacobian. 
>> This appears particularly useful for implementing adjoints.
>> 
>> 7) If we added the two new interfaces the internals of an already
>> overly complicated TS become even more convoluted and unmaintainable.  
> 
> We could split the shift into ashift and bshift (elided as always 1.0
> now), then users could opt to skip work if one of those was nonzero.
> That would be a modest generalization from what we have now and would
> enable any desired optimization.  Integrators that wish to take
> advantage of M not changing could interpret a flag saying that it was
> constant and then always call the IJacobian with ashift=0.  That would
> be unintrusive for existing users and still enable all optimizations.
> It's only one additional parameter and then optimized user code could
> look like
> 
>  for (each element) {
>Ke[m][m] = {};
>if (ashift != 0) {
>  Ke += ashift * (mass stuff);
>}
>if (bshift != 0) {
>  Ke += bshift * (non-mass stuff);
>}
>  }
> 
> Naive user code would elide the conditionals, but would still work with
> all integrators.

   So this is a backdoor way of allowing the user to provide separate functions 
for computing the mass matrix and the Jacobian but the current TSSetIJacobian() 
function only takes one set of matrices in which means that if it returns just 
the mass matrix it returns it with the storage pattern of the Jacobian, so if 
the mass matrix is diagonal you get back a sparse matrix with many explicit 
nonzeros and just the entries on the diagonal. 

   How about the following based on suggestions from several people.

 Keep the current public API TSSetIJacobian() as is

  Add two new public functions:TSSetMass(Mat, functionmass),  
TSSetIJacobianWithOutMass(Mat, functionwithoutmass) (bad name).

  Provide developer functionsTSComputeMass() which uses functionmass when 
possible else ijacobian, TSComputeIJacobianWithoutMass() that computes using 
functionwithoutmass if possible otherwise ijacobian. And change 
TSComputeIJacobian() so it uses i jacobian if it exists otherwise uses 
functionmass and functionwithoutmass and combines them automatically. 

  Drawbacks include more complicated internal code (though with the helper 
functions it will be cleaner especially in the adjoints), power users can 
provide combined operation, non power users have a simpler interface. 

Re: [petsc-dev] Proposed changes to TS API

[Smith, Barry F.]

> On May 12, 2018, at 1:17 PM, Jed Brown  wrote:
> 
> "Smith, Barry F."  writes:
> 
>>> On May 11, 2018, at 6:05 PM, Jed Brown  wrote:
>>> 
>>> "Smith, Barry F."  writes:
>>> 
>>>>  Here is MY summary of the discussion so far.
>>>> 
>>>> 1) the IFunction/IJacobian interface has its supporters. There is valid 
>>>> argument that for certain cases it can be more efficient than the proposed 
>>>> alternative; but this seems largely a theoretical believe at this time 
>>>> since there are no comparisons between nontrivial (or even trivial) codes 
>>>> that use the assembly directly the mass shift plus Jacobian vs the 
>>>> assembly separately and MatAXPY the two parts together.  As far as I can 
>>>> see this potential performance is the ONLY benefit to keeping the 
>>>> IFunction/IJacobian() interface? Please list additional benefits if there 
>>>> are any?
>>> 
>>> PCShell
>>> 
>>>> 2) The IFunction/IJacobian approach makes it impossible for TS to access 
>>>> the mass matrix alone. 
>>> 
>>> Without wasted work, yes.
>>> 
>>>> 3) But one can access the IJacobian matrix alone by passing a shift of zero
>>>> 
>>>> 4) TSComputeIJacobian() is an ugly piece of shit code that has a confusing 
>>>> name since it also can incorporates the RHS Jacobian. 
>>> 
>>> If you get rid of that, then every implicit integrator will need to
>>> handle the RHSFunction/RHSJacobian itself.  It will be significantly
>>> more code to maintain.
>>> 
>>>> 4a) the TSComputeIJacobian() has issues for linear problems because it 
>>>> overwrites the user provided Jacobian and has hacks to deal with it
>>> 
>>> Yes, however that choice reduces memory usage which is sometimes an
>>> important factor.
>>> 
>>>> 5) There is no standard way to solve M u = F() explicitly from the 
>>>> IFunction() form, (and cannot even with expressed with the explicit RHS 
>>>> approach) the user must manage the M solve themselves inside their 
>>>> RHSFunction.
>>> 
>>> We could write this as an IMEX method with IFunction providing M u and
>>> RHSFunction providing F.  I think this could be specialized for constant
>>> M and provided automatically for any explicit method.
>>> 
>>>> 6) There is some support for adding two new function callbacks that a) 
>>>> compute the mass matrix and b) compute the Jacobian part of the IJacobian. 
>>>> This appears particularly useful for implementing adjoints.
>>>> 
>>>> 7) If we added the two new interfaces the internals of an already
>>>> overly complicated TS become even more convoluted and unmaintainable.  
>>> 
>>> We could split the shift into ashift and bshift (elided as always 1.0
>>> now), then users could opt to skip work if one of those was nonzero.
>>> That would be a modest generalization from what we have now and would
>>> enable any desired optimization.  Integrators that wish to take
>>> advantage of M not changing could interpret a flag saying that it was
>>> constant and then always call the IJacobian with ashift=0.  That would
>>> be unintrusive for existing users and still enable all optimizations.
>>> It's only one additional parameter and then optimized user code could
>>> look like
>>> 
>>> for (each element) {
>>>   Ke[m][m] = {};
>>>   if (ashift != 0) {
>>> Ke += ashift * (mass stuff);
>>>   }
>>>   if (bshift != 0) {
>>> Ke += bshift * (non-mass stuff);
>>>   }
>>> }
>>> 
>>> Naive user code would elide the conditionals, but would still work with
>>> all integrators.
>> 
>>   So this is a backdoor way of allowing the user to provide separate 
>> functions for computing the mass matrix and the Jacobian but the current 
>> TSSetIJacobian() function only takes one set of matrices in which means that 
>> if it returns just the mass matrix it returns it with the storage pattern of 
>> the Jacobian, so if the mass matrix is diagonal you get back a sparse matrix 
>> with many explicit nonzeros and just the entries on the diagonal. 
> 
> Okay, that's messy to work around and relevant to methods like compact
> FD where the mass matrix is significantly sparser than the RHS operator.
> 
>>   How about the following based on suggest

Re: [petsc-dev] Nested XML logging and stages

[Smith, Barry F.]

  I am fine with the suggested changes. 

> On May 13, 2018, at 1:06 PM, Lisandro Dalcin  wrote:
> 
> It took me a while to figure out why my xml -log_view was empty. Nested XML
> logging and push/pop user-defined stages simply does not work, the
> implementation cannot cope with this usage pattern.
> 
> Would it be OK to at ignore push/pop stages if XML is on, and then log
> everything in the main stage, to at least get an non-empty xml log file? I
> guess this would require a private global flag.
> 
> Now that I am on it, could we add a command line option to set the XML
> logging threshold? I would like to pass -log_threshold 0 to get everything,
> but surprisingly "0" is not a good value, NaNs are generated. So I guess we
> need something like PetscMax(threshold,1e-10).
> 
> Why PetscLogView_Nested() calls PetscLogNestedEnd() ? While convenient and
> quick to implement, it smells like bad API usage.

  I don't see a nice alternative.

   Barry

> 
> -- 
> Lisandro Dalcin
> 
> Research Scientist
> Computer, Electrical and Mathematical Sciences & Engineering (CEMSE)
> Extreme Computing Research Center (ECRC)
> King Abdullah University of Science and Technology (KAUST)
> http://ecrc.kaust.edu.sa/
> 
> 4700 King Abdullah University of Science and Technology
> al-Khawarizmi Bldg (Bldg 1), Office # 0109
> Thuwal 23955-6900, Kingdom of Saudi Arabia
> http://www.kaust.edu.sa
> 
> Office Phone: +966 12 808-0459

Re: [petsc-dev] Problem with l2g IS created by VecCreateGhostBlock

[Smith, Barry F.]

   Garth,

Sorry for the delay in responding. This is our bug, I have put the fix into 
the branch barry/fix-veccreateghostblockwitharray/maint
which after testing tonight will go into the maint branch and then the next 
patch release.

 Please let us know if the fix in the branch resolves your difficulties.

Barry


> On May 17, 2018, at 11:41 AM, Garth Wells  wrote:
> 
> I'm seeing an issue because I previously built my own l2g IS, whereas
> now I'm relying on the IS created by VecCreateGhostBlock.
> 
> For a vector with 8 entries and block size 2 (i.e, 4 blocks), and
> arranged:
> 
> P0: Owns blocks: 0
>Ghost blocks: 1, 2 
> 
> P1: Owns blocks: 1, 2, 3
>Ghost blocks: None
> 
> I expected the local-to-global IS created by VecCreateGhostBlock on
> proc 1 to be:
> 
> 0  1
> 1  2
> 2  3
> 
> But I get 
> 
> 0  2  
> 1  3
> 2  4
> 
> The below code illustrates (run with 2 procs). I'm using the dev master
> branch (updated just now).
> 
> Am I doing something wrong?
> 
> Garth
> 
> 
> 
> #include 
> static char help[] = "Test ghost vec";
> int main(int argc, char **argv)
> {
>  PetscInitialize(&argc, &argv, (char *)0, help);
> 
>  PetscMPIInt rank;
>  MPI_Comm_rank(PETSC_COMM_WORLD, &rank);
> 
>  Vec x;
>  if (rank == 0)
>  {
>PetscInt ghst[2] = {1, 2};
>VecCreateGhostBlock(PETSC_COMM_WORLD, 2, 2, PETSC_DECIDE, 2, ghst,
> &x);
>  }
>  else if (rank == 1)
>  {
>VecCreateGhostBlock(PETSC_COMM_WORLD, 2, 6, PETSC_DECIDE, 0, NULL,
> &x);
> 
>ISLocalToGlobalMapping mapping;
>VecGetLocalToGlobalMapping(x, &mapping);
>ISLocalToGlobalMappingView(mapping, PETSC_VIEWER_STDOUT_SELF);
>  }
>  return 0;
> }

Re: [petsc-dev] Mat updating from 3.7 to 3.9.2 Fortran

[Smith, Barry F.]

 Use -mat_view on the new and old code to verify that the same matrix is 
actually being generated.

   Barry


> On May 23, 2018, at 6:18 PM, Hector E Barrios Molano  
> wrote:
> 
> Thanks Jed for the Answer.
> 
> I am still having problems with this code. In summary what I changed from 
> PETSc 3.7 to 3.9 was:
> 
> - use of PETSC_NULL_MAT to evaluate if the matrix was defined
> 
> mat = PETSC_NULL_MAT
> ...
> if(mat .eq. PETSC_NULL_MAT)then
> create matrix,vectors, ksp
> else
> KSPSetInitialGuessNonzero
> VecResetArray
> end if
> 
> - modified MatSetValuesBlocked call
> Previously it was:
> 
> call MatSetValuesBlocked(mat,1,N-1,IROW,LC(nnz),BB(nnz),INSERT_VALUES,ierr)
> 
> with N-1, LC(nnz), and BB(nnz) scalar values, the elements where inserted one 
> at a time
> 
> Modified to:
> 
> call 
> MatSetValuesBlocked(mat,1,[N-1],IROW,[LC(nnz)],[BB(nnz)],INSERT_VALUES,ierr)
> 
> After the matrix is built there is a call to 
> 
> KSPSolve
> 
> When I run the program I get a lot of these errors:
> 
> [0]PETSC ERROR: - Error Message 
> --
> [0]PETSC ERROR: Object is in wrong state
> [0]PETSC ERROR: Matrix is missing diagonal entry 1
> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for 
> trouble shooting.
> [0]PETSC ERROR: Petsc Release Version 3.9.2, unknown 
> [0]PETSC ERROR: ../../../UTCOMPRS on a  named bandera by hector Wed May 23 
> 18:08:07 2018
> [0]PETSC ERROR: Configure options 
> --prefix=/home/hector/installed/petsc_git-intel-debug 
> --PETSC_DIR=/home/hector/dwnld_prog/petsc --PETSC_ARCH=linux-intel-debug 
> --CC=mpiicc --FC=mpiifort --CXX=mpiicpc --with-openmp=1 --with-valgrind=1 
> --with-valgrind-dir=/home/hector/installed 
> --with-parmetis-dir=/home/hector/installed/parmetis/ 
> --with-metis-dir=/home/hector/installed/parmetis/ 
> --with-zoltan-dir=/home/hector/installed/zoltan/ 
> --with-hypre-dir=/home/hector/installed/hypre --download-ptscotch 
> --with-blaslapack-lib="[/home/hector/installed/intel/compilers_and_libraries_2019.0.046/linux/mkl/lib/intel64/libmkl_intel_lp64.a,/home/hector/installed/intel/compilers_and_libraries_2019.0.046/linux/mkl/lib/intel64/libmkl_core.a,/home/hector/installed/intel/compilers_and_libraries_2019.0.046/linux/mkl/lib/intel64/libmkl_intel_thread.a]"
>  
> --with-scalapack-include=/home/hector/installed/intel/compilers_and_libraries_2019.0.046/linux/mkl/include
>  
> --with-scalapack-lib="[/home/hector/installed/intel/compilers_and_libraries_2019.0.046/linux/mkl/lib/intel64/libmkl_scalapack_lp64.a,/home/hector/installed/intel/compilers_and_libraries_2019.0.046/linux/mkl/lib/intel64/libmkl_blacs_intelmpi_lp64.a]"
>  --with-shared-libraries=0 --FC_LINKER_FLAGS="-qopenmp -qopenmp-link static" 
> --FFLAGS="-qopenmp -qopenmp-link static" --LIBS="-Wl,--start-group 
> /home/hector/installed/intel/compilers_and_libraries_2019.0.046/linux/mkl/lib/intel64/libmkl_intel_lp64.a
>  
> /home/hector/installed/intel/compilers_and_libraries_2019.0.046/linux/mkl/lib/intel64/libmkl_core.a
>  
> /home/hector/installed/intel/compilers_and_libraries_2019.0.046/linux/mkl/lib/intel64/libmkl_intel_thread.a
>  -Wl,--end-group -liomp5 -ldl -lpthread -lm"
> [0]PETSC ERROR: #305869 MatILUFactorSymbolic_SeqBAIJ() line 369 in 
> /home/hector/dwnld_prog/petsc/src/mat/impls/baij/seq/baijfact2.c
> [0]PETSC ERROR: #305870 MatILUFactorSymbolic() line 6522 in 
> /home/hector/dwnld_prog/petsc/src/mat/interface/matrix.c
> [0]PETSC ERROR: #305871 PCSetUp_ILU() line 144 in 
> /home/hector/dwnld_prog/petsc/src/ksp/pc/impls/factor/ilu/ilu.c
> [0]PETSC ERROR: #305872 PCSetUp() line 923 in 
> /home/hector/dwnld_prog/petsc/src/ksp/pc/interface/precon.c
> [0]PETSC ERROR: #305873 KSPSetUp() line 381 in 
> /home/hector/dwnld_prog/petsc/src/ksp/ksp/interface/itfunc.c
> [0]PETSC ERROR: #305874 KSPSolve() line 612 in 
> /home/hector/dwnld_prog/petsc/src/ksp/ksp/interface/itfunc.c
> 
> 
> What could be the problem?
> 
> Thanks for your help,
> 
> Hector
> 
> 
> 
> On 05/22/2018 06:28 PM, Jed Brown wrote:
>> Hector E Barrios Molano 
>>  writes:
>> 
>> 
>>> Hi PETSc Experts!
>>> 
>>> I am updating a Fortran code that use PETSc 3.7 to version 3.9.2 (from 
>>> git repository).
>>> 
>>> Such code declares:
>>> 
>>> Mat mat
>>> 
>>> and used it as an integer, for example, to assign an initial value and 
>>> test it to know if the matrix has been created by PETSc.
>>> 
>>> data mat/-1/
>>> if (mat .eq. -1) then
>>> 
>> You can use PETSC_NULL_MAT
>> 
>> 
>>> With the new PETSc Fortran modules the compiler complains about this and 
>>> stops.
>>> 
>>> Is there a better way to achieve this? So that I do not have to set an 
>>> predefined value to Mat type to test if it was already created by PETSc?
>>> 
>>> If not, Is there a way to access the value in mat?
>>> Looking at the source code type(tMat) has "v" variable, so could I used 
>>> as mat%v?
>>> 
>> Please no.
>> 
> 

Re: [petsc-dev] changes for petsc tests for XL Fortran compiler

[Smith, Barry F.]

   Serban,

 Thanks for the patches. We'll get fixes into the maint branch and hence 
the next patch release.

   Satish,

 We have the macro PetscFlush() that should be used for the call flush(6) 
locations to make the Fortran example code portable. (But needs to be tested 
since currently it is not used).

Barry


> On Jun 1, 2018, at 9:21 AM, Serban Maerean  wrote:
> 
> Hello,
> 
> FYI, I needed to build PETSc test cases with the XL Fortran compiler and had 
> to make the following changes, in  directory>/share/petsc/examples/: .
> 
> The reasons for these changes was to be able to build w/the XL Fortran 
> compiler (and w/out Fortran name mangling, which is the default for the XL 
> Fortran compiler.)
> 
> Let me know if you have any questions.
> 
> Thanks,
> 
> 
> Serban Maerean
> HPC Tools
> T/L: 293-9770, Tel.: 845-433-9770
> E-mail: ser...@us.ibm.com 
> 

Re: [petsc-dev] good partitioning packages?

[Smith, Barry F.]

   Fande,

   This is a great question. I am forwarding it to Mike Heroux who has a 
high level position in the ECP; because I have similar concerns and also don't 
have a good answer. ParMetis does indeed have a poor license and essentially no 
support. Perhaps Mike has some ideas.

   Barry



On Jun 1, 2018, at 5:46 PM, Kong, Fande  wrote:

Hi Developers,

I have introduced MatPartitioning interface to MOOSE. It is working great, and 
we can use all external partitioning packages via a simple interface.

But here is a concern. Almost all the packages are not under development any 
more. Does this make a bug fix more difficult in the future.  Also some of them 
have bad licenses.

I was wondering there is any other partitioning package in the community?

Thanks,

Fande,

Re: [petsc-dev] Tiny pull requests via Bitbucket web interface

[Smith, Barry F.]

> On Jun 6, 2018, at 10:06 AM, Karl Rupp  wrote:
> 
> Hi all,
> 
> yes, I support Patrick's idea of actively encouraging such simple pull 
> requests. Particularly when it comes to documentation, it would be very handy 
> to also add a link to the manual pages on the top right. For example,
> http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Mat/MatSetValue.html
> currently states "Report Typos and Errors". We could add another line, e.g. 
> "Fix Typos and Errors via Bitbucket" pointing to the respective source file 
> (maybe even the source line can be pointed to).

Why isn't this in an on-line pull request?


> 
> Best regards,
> Karli
> 
> 
> 
> On 06/05/2018 07:02 PM, Patrick Sanan wrote:
>> After Karl's nice talk on contributing to PETSc, I was reminded of a similar 
>> talk that I saw at the Julia conference. The title was also something like 
>> "Contributing is easy!" but used an even more extreme example of making your 
>> first contribution. It was (as Barry encouraged) a small documentation 
>> change, and they demonstrated how to do this via the GitHub web interface.
>> This could be a great way to lower the "activation energy" for these kinds 
>> of tiny, trivially-reviewable changes.
>> The practical steps with Bitbucket are approximately :
>>  * go to the PETSc Bitbucket site
>>  * navigate to the source file you want to change
>>  * "edit"
>>  * make sure you are at "master" (I had to select this from the
>>pull-down, otherwise "edit" was greyed out and and gives a hint on
>>mouseover)
>>  * make your small, innocuous edit
>>  * "commit"
>>  * select "create a pull request" if needbe, and fill out comments /
>>reviewers as usual.
>> I believe that if you don't have write access, you can still do this and it 
>> will create a fork for you automatically.
>> Here's a test:
>> https://bitbucket.org/petsc/petsc/pull-requests/975/docs-manual-makefile-fix-typo-in-error/diff
>> Thoughts? Should this be actively encouraged?

[petsc-dev] faster and/or parallel build for petsc4py?

[Smith, Barry F.]

   Petsc4py takes a long time to build on my laptop (longer than PETSc itself?) 
It appears to not use a parallel build; any chance it can be changed to use a 
parallel build or to take less time to build?

   Thanks

Barry

Re: [petsc-dev] faster and/or parallel build for petsc4py?

[Smith, Barry F.]

  No way to separate into a bunch of independent small files?

   Barry


> On Jun 6, 2018, at 4:24 PM, Stefano Zampini  wrote:
> 
> It is a simple huge file compiled. No way to do a parallel build
> 
>> On Jun 6, 2018, at 4:19 PM, Smith, Barry F.  wrote:
>> 
>> 
>>  Petsc4py takes a long time to build on my laptop (longer than PETSc 
>> itself?) It appears to not use a parallel build; any chance it can be 
>> changed to use a parallel build or to take less time to build?
>> 
>>  Thanks
>> 
>>   Barry
>> 
> 

Re: [petsc-dev] [petsc-users] Poor weak scaling when solving successive linearsystems

[Smith, Barry F.]

lculating the 
> electric potential from a homogeneous charge distribution, done multiple 
> times to accumulate time in KSPsolve().
> A job would be started using something like
> mpirun -n 125 ~/petsc_ws/ws_test -nodes_per_proc 30 -mesh_size 1E-4 
> -iterations 1000 \\
> -ksp_rtol 1E-6 \
> -log_view -log_sync\
> -pc_type gamg -pc_gamg_type classical\
> -ksp_type cg \
> -ksp_norm_type unpreconditioned \
> -mg_levels_ksp_type richardson \
> -mg_levels_ksp_norm_type none \
> -mg_levels_pc_type sor \
> -mg_levels_ksp_max_it 1 \
> -mg_levels_pc_sor_its 1 \
> -mg_levels_esteig_ksp_type cg \
> -mg_levels_esteig_ksp_max_it 10 \
> -gamg_est_ksp_type cg
> , ideally started on a cube number of processes for a cubical process grid.
> Using 125 processes and 10.000 iterations I get the output in 
> "log_view_125_new.txt", which shows the same imbalance for me.
> Michael
> 
> 
> Am 02.06.2018 um 13:40 schrieb Mark Adams:
>> 
>> 
>> On Fri, Jun 1, 2018 at 11:20 PM, Junchao Zhang  wrote:
>> Hi,Michael,
>>  You can add -log_sync besides -log_view, which adds barriers to certain 
>> events but measures barrier time separately from the events. I find this 
>> option makes it easier to interpret log_view output.
>> 
>> That is great (good to know).
>> 
>> This should give us a better idea if your large VecScatter costs are from 
>> slow communication or if it catching some sort of load imbalance.
>> 
>> 
>> --Junchao Zhang
>> 
>> On Wed, May 30, 2018 at 3:27 AM, Michael Becker 
>>  wrote:
>> Barry: On its way. Could take a couple days again.
>> 
>> Junchao: I unfortunately don't have access to a cluster with a faster 
>> network. This one has a mixed 4X QDR-FDR InfiniBand 2:1 blocking fat-tree 
>> network, which I realize causes parallel slowdown if the nodes are not 
>> connected to the same switch. Each node has 24 processors (2x12/socket) and 
>> four NUMA domains (two for each socket).
>> The ranks are usually not distributed perfectly even, i.e. for 125 
>> processes, of the six required nodes, five would use 21 cores and one 20.
>> Would using another CPU type make a difference communication-wise? I could 
>> switch to faster ones (on the same network), but I always assumed this would 
>> only improve performance of the stuff that is unrelated to communication.
>> 
>> Michael
>> 
>> 
>> 
>>> The log files have something like "Average time for zero size MPI_Send(): 
>>> 1.84231e-05". It looks you ran on a cluster with a very slow network. A 
>>> typical machine should give less than 1/10 of the latency you have. An easy 
>>> way to try is just running the code on a machine with a faster network and 
>>> see what happens.
>>> 
>>> Also, how many cores & numa domains does a compute node have? I could not 
>>> figure out how you distributed the 125 MPI ranks evenly.
>>> 
>>> --Junchao Zhang
>>> 
>>> On Tue, May 29, 2018 at 6:18 AM, Michael Becker 
>>>  wrote:
>>> Hello again,
>>> 
>>> here are the updated log_view files for 125 and 1000 processors. I ran both 
>>> problems twice, the first time with all processors per node allocated 
>>> ("-1.txt"), the second with only half on twice the number of nodes 
>>> ("-2.txt").
>>> 
>>>>> On May 24, 2018, at 12:24 AM, Michael Becker 
>>>>> 
>>>>> wrote:
>>>>> 
>>>>> I noticed that for every individual KSP iteration, six vector objects are 
>>>>> created and destroyed (with CG, more with e.g. GMRES). 
>>>>> 
>>>>   Hmm, it is certainly not intended at vectors be created and destroyed 
>>>> within each KSPSolve() could you please point us to the code that makes 
>>>> you think they are being created and destroyed?   We create all the work 
>>>> vectors at KSPSetUp() and destroy them in KSPReset() not during the solve. 
>>>> Not that this would be a measurable distance.
>>>> 
>>> 
>>> I mean this, right in the log_view output:
>>> 
>>>> Memory usage is given in bytes:
>>>> 
>>>> Object Type Creations Destructions Memory Descendants' Mem.
>>>> Reports information only for process 0.
>>>> 
>>>> --- Event Stage 0: Main Stage
>>>> 
>>>> ...
>>>> 
>>>> --- Event Stage 1: First Solve
>>>> 
>>>> ...
>>>> 
>>>> --- E

Re: [petsc-dev] [petsc-users] Poor weak scaling when solving successive linearsystems

[Smith, Barry F.]

lculating the 
> electric potential from a homogeneous charge distribution, done multiple 
> times to accumulate time in KSPsolve().
> A job would be started using something like
> mpirun -n 125 ~/petsc_ws/ws_test -nodes_per_proc 30 -mesh_size 1E-4 
> -iterations 1000 \\
> -ksp_rtol 1E-6 \
> -log_view -log_sync\
> -pc_type gamg -pc_gamg_type classical\
> -ksp_type cg \
> -ksp_norm_type unpreconditioned \
> -mg_levels_ksp_type richardson \
> -mg_levels_ksp_norm_type none \
> -mg_levels_pc_type sor \
> -mg_levels_ksp_max_it 1 \
> -mg_levels_pc_sor_its 1 \
> -mg_levels_esteig_ksp_type cg \
> -mg_levels_esteig_ksp_max_it 10 \
> -gamg_est_ksp_type cg
> , ideally started on a cube number of processes for a cubical process grid.
> Using 125 processes and 10.000 iterations I get the output in 
> "log_view_125_new.txt", which shows the same imbalance for me.
> Michael
> 
> 
> Am 02.06.2018 um 13:40 schrieb Mark Adams:
>> 
>> 
>> On Fri, Jun 1, 2018 at 11:20 PM, Junchao Zhang  wrote:
>> Hi,Michael,
>>  You can add -log_sync besides -log_view, which adds barriers to certain 
>> events but measures barrier time separately from the events. I find this 
>> option makes it easier to interpret log_view output.
>> 
>> That is great (good to know).
>> 
>> This should give us a better idea if your large VecScatter costs are from 
>> slow communication or if it catching some sort of load imbalance.
>> 
>> 
>> --Junchao Zhang
>> 
>> On Wed, May 30, 2018 at 3:27 AM, Michael Becker 
>>  wrote:
>> Barry: On its way. Could take a couple days again.
>> 
>> Junchao: I unfortunately don't have access to a cluster with a faster 
>> network. This one has a mixed 4X QDR-FDR InfiniBand 2:1 blocking fat-tree 
>> network, which I realize causes parallel slowdown if the nodes are not 
>> connected to the same switch. Each node has 24 processors (2x12/socket) and 
>> four NUMA domains (two for each socket).
>> The ranks are usually not distributed perfectly even, i.e. for 125 
>> processes, of the six required nodes, five would use 21 cores and one 20.
>> Would using another CPU type make a difference communication-wise? I could 
>> switch to faster ones (on the same network), but I always assumed this would 
>> only improve performance of the stuff that is unrelated to communication.
>> 
>> Michael
>> 
>> 
>> 
>>> The log files have something like "Average time for zero size MPI_Send(): 
>>> 1.84231e-05". It looks you ran on a cluster with a very slow network. A 
>>> typical machine should give less than 1/10 of the latency you have. An easy 
>>> way to try is just running the code on a machine with a faster network and 
>>> see what happens.
>>> 
>>> Also, how many cores & numa domains does a compute node have? I could not 
>>> figure out how you distributed the 125 MPI ranks evenly.
>>> 
>>> --Junchao Zhang
>>> 
>>> On Tue, May 29, 2018 at 6:18 AM, Michael Becker 
>>>  wrote:
>>> Hello again,
>>> 
>>> here are the updated log_view files for 125 and 1000 processors. I ran both 
>>> problems twice, the first time with all processors per node allocated 
>>> ("-1.txt"), the second with only half on twice the number of nodes 
>>> ("-2.txt").
>>> 
>>>>> On May 24, 2018, at 12:24 AM, Michael Becker 
>>>>> 
>>>>> wrote:
>>>>> 
>>>>> I noticed that for every individual KSP iteration, six vector objects are 
>>>>> created and destroyed (with CG, more with e.g. GMRES). 
>>>>> 
>>>>   Hmm, it is certainly not intended at vectors be created and destroyed 
>>>> within each KSPSolve() could you please point us to the code that makes 
>>>> you think they are being created and destroyed?   We create all the work 
>>>> vectors at KSPSetUp() and destroy them in KSPReset() not during the solve. 
>>>> Not that this would be a measurable distance.
>>>> 
>>> 
>>> I mean this, right in the log_view output:
>>> 
>>>> Memory usage is given in bytes:
>>>> 
>>>> Object Type Creations Destructions Memory Descendants' Mem.
>>>> Reports information only for process 0.
>>>> 
>>>> --- Event Stage 0: Main Stage
>>>> 
>>>> ...
>>>> 
>>>> --- Event Stage 1: First Solve
>>>> 
>>>> ...
>>>> 
>>>> --- E

Re: [petsc-dev] [petsc-users] Poor weak scaling when solving successive linearsystems

[Smith, Barry F.]

> On Jun 7, 2018, at 12:27 PM, Zhang, Junchao  wrote:
> 
> Searched but could not find this option, -mat_view::load_balance

   There is a space between the view and the :   load_balance is a particular 
viewer format that causes the printing of load balance information about number 
of nonzeros in the matrix.

   Barry

> 
> --Junchao Zhang
> 
> On Thu, Jun 7, 2018 at 10:46 AM, Smith, Barry F.  wrote:
>  So the only surprise in the results is the SOR. It is embarrassingly 
> parallel and normally one would not see a jump.
> 
>  The load balance for SOR time 1.5 is better at 1000 processes than for 125 
> processes of 2.1  not worse so this number doesn't easily explain it.
> 
>  Could you run the 125 and 1000 with -mat_view ::load_balance and see what 
> you get out?
> 
>Thanks
> 
>  Barry
> 
>  Notice that the MatSOR time jumps a lot about 5 secs when the -log_sync is 
> on. My only guess is that the MatSOR is sharing memory bandwidth (or some 
> other resource? cores?) with the VecScatter and for some reason this is worse 
> for 1000 cores but I don't know why.
> 
> > On Jun 6, 2018, at 9:13 PM, Junchao Zhang  wrote:
> > 
> > Hi, PETSc developers,
> >  I tested Michael Becker's code. The code calls the same KSPSolve 1000 
> > times in the second stage and needs cubic number of processors to run. I 
> > ran with 125 ranks and 1000 ranks, with or without -log_sync option. I 
> > attach the log view output files and a scaling loss excel file.
> >  I profiled the code with 125 processors. It looks {MatSOR, MatMult, 
> > MatMultAdd, MatMultTranspose, MatMultTransposeAdd}_SeqAIJ in aij.c took 
> > ~50% of the time,  The other half time was spent on waiting in MPI.  
> > MatSOR_SeqAIJ took 30%, mostly in PetscSparseDenseMinusDot().
> >  I tested it on a 36 cores/node machine. I found 32 ranks/node gave better 
> > performance (about 10%) than 36 ranks/node in the 125 ranks testing.  I 
> > guess it is because processors in the former had more balanced memory 
> > bandwidth. I collected PAPI_DP_OPS (double precision operations) and 
> > PAPI_TOT_CYC (total cycles) of the 125 ranks case (see the attached files). 
> > It looks ranks at the two ends have less DP_OPS and TOT_CYC. 
> >  Does anyone familiar with the algorithm have quick explanations?
> > 
> > --Junchao Zhang
> > 
> > On Mon, Jun 4, 2018 at 11:59 AM, Michael Becker 
> >  wrote:
> > Hello again,
> > 
> > this took me longer than I anticipated, but here we go.
> > I did reruns of the cases where only half the processes per node were used 
> > (without -log_sync):
> > 
> > 125 procs,1st   125 procs,2nd  1000 
> > procs,1st  1000 procs,2nd
> >   MaxRatioMaxRatioMax   
> >  RatioMaxRatio
> > KSPSolve   1.203E+021.01.210E+021.0
> > 1.399E+021.11.365E+021.0
> > VecTDot6.376E+003.76.551E+004.0
> > 7.885E+002.97.175E+003.4
> > VecNorm4.579E+007.15.803E+00   10.2
> > 8.534E+006.96.026E+004.9
> > VecScale   1.070E-012.11.129E-012.2
> > 1.301E-012.51.270E-012.4
> > VecCopy1.123E-011.31.149E-011.3
> > 1.301E-011.61.359E-011.6
> > VecSet 7.063E-011.76.968E-011.7
> > 7.432E-011.87.425E-011.8
> > VecAXPY1.166E+001.41.167E+001.4
> > 1.221E+001.51.279E+001.6
> > VecAYPX1.317E+001.61.290E+001.6
> > 1.536E+001.91.499E+002.0
> > VecScatterBegin6.142E+003.25.974E+002.8
> > 6.448E+003.06.472E+002.9
> > VecScatterEnd  3.606E+014.23.551E+014.0
> > 5.244E+012.74.995E+012.7
> > MatMult3.561E+011.63.403E+011.5
> > 3.435E+011.43.332E+011.4
> > MatMultAdd 1.124E+012.01.130E+012.1
> > 2.093E+012.91.995E+012.7
> > MatMultTranspose   1.372E+012.51.388E+012.6
> > 1.477E+012.21.381E+012.1
> > MatSolve   1.949E-020.01.653E-020.0
> > 4.789E-020.04.466E-020.0
> > MatSOR 6.610E+011.36.673E+011.3
> > 7.

Re: [petsc-dev] [petsc-users] Poor weak scaling when solving successive linearsystems

[Smith, Barry F.]

  Junchao,

  Thanks, the load balance of matrix entries is remarkably similar for the 
two runs so it can't be a matter of worse work load imbalance for SOR for the 
larger case explaining why the SOR takes more time. 

  Here is my guess (and I know no way to confirm it). In the smaller case 
the overlap of different processes on the same node running SOR at the same 
time is lower than the larger case hence the larger case is slower because 
there are more SOR processes fighting over the same memory bandwidth at the 
same time than in the smaller case.   Ahh, here is something you can try, lets 
undersubscribe the memory bandwidth needs, run on say 16 processes per node 
with 8 nodes and 16 processes per node with 64 nodes and send the two -log_view 
output files. I assume this is an LCRC machine and NOT a KNL system?

   Thanks


   Barry


> On Jun 9, 2018, at 8:29 AM, Mark Adams  wrote:
> 
> -pc_gamg_type classical
> 
> FYI, we only support smoothed aggregation "agg" (the default). (This thread 
> started by saying you were using GAMG.)
> 
> It is not clear how much this will make a difference for you, but you don't 
> want to use classical because we do not support it. It is meant as a 
> reference implementation for developers.
> 
> First, how did you get the idea to use classical? If the documentation lead 
> you to believe this was a good thing to do then we need to fix that!
> 
> Anyway, here is a generic input for GAMG:
> 
> -pc_type gamg 
> -pc_gamg_type agg 
> -pc_gamg_agg_nsmooths 1 
> -pc_gamg_coarse_eq_limit 1000 
> -pc_gamg_reuse_interpolation true 
> -pc_gamg_square_graph 1 
> -pc_gamg_threshold 0.05 
> -pc_gamg_threshold_scale .0 
> 
> 
> 
> 
> On Thu, Jun 7, 2018 at 6:52 PM, Junchao Zhang  wrote:
> OK, I have thought that space was a typo. btw, this option does not show up 
> in -h.
> I changed number of ranks to use all cores on each node to avoid misleading 
> ratio in -log_view. Since one node has 36 cores, I ran with 6^3=216 ranks, 
> and 12^3=1728 ranks. I also found call counts of MatSOR etc in the two tests 
> were different. So they are not strict weak scaling tests. I tried to add 
> -ksp_max_it 6 -pc_mg_levels 6, but still could not make the two have the same 
> MatSOR count. Anyway, I attached the load balance output.
> 
> I find PCApply_MG calls PCMGMCycle_Private, which is recursive and indirectly 
> calls MatSOR_MPIAIJ. I believe the following code in MatSOR_MPIAIJ 
> practically syncs {MatSOR, MatMultAdd}_SeqAIJ  between processors through 
> VecScatter at each MG level. If SOR and MatMultAdd are imbalanced, the cost 
> is accumulated along MG levels and shows up as large VecScatter cost.
> 1460: while
>  (its--) {
> 
> 1461:   
> VecScatterBegin(mat->Mvctx,xx,mat->lvec,INSERT_VALUES,SCATTER_FORWARD
> );
> 
> 1462:   
> VecScatterEnd(mat->Mvctx,xx,mat->lvec,INSERT_VALUES,SCATTER_FORWARD
> );
> 
> 
> 1464:   /* update rhs: bb1 = bb - B*x */
> 1465:   VecScale
> (mat->lvec,-1.0);
> 
> 1466:   (*mat->B->ops->multadd)(mat->
> B,mat->lvec,bb,bb1);
> 
> 
> 1468:   /* local sweep */
> 1469:   (*mat->A->ops->sor)(mat->A,bb1,omega,SOR_SYMMETRIC_SWEEP,
> fshift,lits,1,xx);
> 
> 1470: }
> 
> 
> 
> --Junchao Zhang
> 
> On Thu, Jun 7, 2018 at 3:11 PM, Smith, Barry F.  wrote:
> 
> 
> > On Jun 7, 2018, at 12:27 PM, Zhang, Junchao  wrote:
> > 
> > Searched but could not find this option, -mat_view::load_balance
> 
>There is a space between the view and the :   load_balance is a particular 
> viewer format that causes the printing of load balance information about 
> number of nonzeros in the matrix.
> 
>Barry
> 
> > 
> > --Junchao Zhang
> > 
> > On Thu, Jun 7, 2018 at 10:46 AM, Smith, Barry F.  wrote:
> >  So the only surprise in the results is the SOR. It is embarrassingly 
> > parallel and normally one would not see a jump.
> > 
> >  The load balance for SOR time 1.5 is better at 1000 processes than for 125 
> > processes of 2.1  not worse so this number doesn't easily explain it.
> > 
> >  Could you run the 125 and 1000 with -mat_view ::load_balance and see what 
> > you get out?
> > 
> >Thanks
> > 
> >  Barry
> > 
> >  Notice that the MatSOR time jumps a lot about 5 secs when the -log_sync is 
> > on. My only guess is that the MatSOR is sharing memory bandwidth (or some 
> > other resource? cores?) with the VecScatter and for some reason this is 
> > worse for 1000 cores but I don't know why.
> > 
> > > On Jun 6, 2018, at 9:13 PM, Junchao Zhang 

Re: [petsc-dev] superlu_dist: update to version v5.4.0

[Smith, Barry F.]

  Complain to Sherry and have her fix it before moving the new release into 
PETSc

   Barry


> On Jun 12, 2018, at 10:59 AM, Satish Balay  wrote:
> 
> 
> $ diff superlu/SRC/superlu_enum_consts.h 
> superlu_dist/SRC/superlu_enum_consts.h
> 16a17
>> * January 28, 2018
> 28c29
> < typedef enum {NOROWPERM, LargeDiag, MY_PERMR}   rowperm_t;
> ---
>> typedef enum {NOROWPERM, LargeDiag_MC64, LargeDiag_AWPM, MY_PERMR} rowperm_t;
> 34c35,36
> < typedef enum {LUSUP, UCOL, LSUB, USUB, LLVL, ULVL}  MemType;
> ---
>> //typedef enum {LUSUP, UCOL, LSUB, USUB, LLVL, ULVL, NO_MEMTYPE}  MemType;
>> typedef enum {USUB, LSUB, UCOL, LUSUP, LLVL, ULVL, NO_MEMTYPE}  MemType;
> 69a72,74
>>COMM_DIAG, /* Bcast diagonal block to process column */
>>COMM_RIGHT, /* communicate L panel */
>>COMM_DOWN, /* communicate U panel */
> 
> So superlu and superlu_dist share the same 'superlu_enum_consts.h'
> file - but with differences that are now incompatible with each other.
> 
> I'm not sure how to resolve this..
> 
> Satish
> 
> On Tue, 12 Jun 2018, PETSc checkBuilds wrote:
> 
>> 
>> 
>> Dear PETSc developer,
>> 
>> This email contains listings of contributions attributed to you by
>> `git blame` that caused compiler errors or warnings in PETSc automated
>> testing.  Follow the links to see the full log files. Please attempt to fix
>> the issues promptly or let us know at petsc-dev@mcs.anl.gov if you are unable
>> to resolve the issues.
>> 
>> Thanks,
>>  The PETSc development team
>> 
>> 
>> 
>> warnings attributed to commit 
>> https://bitbucket.org/petsc/petsc/commits/8e05559
>> superlu_dist: update to version v5.4.0
>> 
>>  src/mat/impls/aij/mpi/superlu_dist/superlu_dist.c:571
>>
>> [http://ftp.mcs.anl.gov/pub/petsc/nightlylogs/archive/2018/06/12/build_next_arch-freebsd-cxx-pkgs-opt_wii.log]
>>  
>> /usr/home/balay/petsc.next-2/src/mat/impls/aij/mpi/superlu_dist/superlu_dist.c:571:8:
>>  error: 'LargeDiag_MC64' was not declared in this scope
>>
>> [http://ftp.mcs.anl.gov/pub/petsc/nightlylogs/archive/2018/06/12/build_next_arch-linux-pkgs-cxx-mlib_el6.log]
>>  
>> /home/sandbox/petsc/petsc.next-3/src/mat/impls/aij/mpi/superlu_dist/superlu_dist.c:571:8:
>>  error: 'LargeDiag_MC64' was not declared in this scope
>>
>> [http://ftp.mcs.anl.gov/pub/petsc/nightlylogs/archive/2018/06/12/build_next_arch-osx-10.6-cxx-cmplx-pkgs-dbg_ipro.log]
>>  
>> /Users/petsc/petsc.next-3/src/mat/impls/aij/mpi/superlu_dist/superlu_dist.c:571:8:
>>  error: use of undeclared identifier 'LargeDiag_MC64'
>> 
>>  src/mat/impls/aij/mpi/superlu_dist/superlu_dist.c:574
>>
>> [http://ftp.mcs.anl.gov/pub/petsc/nightlylogs/archive/2018/06/12/build_next_arch-freebsd-cxx-pkgs-opt_wii.log]
>>  
>> /usr/home/balay/petsc.next-2/src/mat/impls/aij/mpi/superlu_dist/superlu_dist.c:574:8:
>>  error: 'LargeDiag_AWPM' was not declared in this scope
>>
>> [http://ftp.mcs.anl.gov/pub/petsc/nightlylogs/archive/2018/06/12/build_next_arch-linux-pkgs-cxx-mlib_el6.log]
>>  
>> /home/sandbox/petsc/petsc.next-3/src/mat/impls/aij/mpi/superlu_dist/superlu_dist.c:574:8:
>>  error: 'LargeDiag_AWPM' was not declared in this scope
>>
>> [http://ftp.mcs.anl.gov/pub/petsc/nightlylogs/archive/2018/06/12/build_next_arch-osx-10.6-cxx-cmplx-pkgs-dbg_ipro.log]
>>  
>> /Users/petsc/petsc.next-3/src/mat/impls/aij/mpi/superlu_dist/superlu_dist.c:574:8:
>>  error: use of undeclared identifier 'LargeDiag_AWPM'
>> 
>>  src/mat/impls/aij/mpi/superlu_dist/superlu_dist.c:720
>>
>> [http://ftp.mcs.anl.gov/pub/petsc/nightlylogs/archive/2018/06/12/build_next_arch-linux-pkgs-cxx-mlib_el6.log]
>>  
>> /home/sandbox/petsc/petsc.next-3/src/mat/impls/aij/mpi/superlu_dist/superlu_dist.c:720:25:
>>  error: 'LargeDiag_MC64' was not declared in this scope
>>
>> [http://ftp.mcs.anl.gov/pub/petsc/nightlylogs/archive/2018/06/12/build_next_arch-osx-10.6-cxx-cmplx-pkgs-dbg_ipro.log]
>>  
>> /Users/petsc/petsc.next-3/src/mat/impls/aij/mpi/superlu_dist/superlu_dist.c:720:25:
>>  error: use of undeclared identifier 'LargeDiag_MC64'
>>
>> [http://ftp.mcs.anl.gov/pub/petsc/nightlylogs/archive/2018/06/12/build_next_arch-freebsd-cxx-pkgs-opt_wii.log]
>>  
>> /usr/home/balay/petsc.next-2/src/mat/impls/aij/mpi/superlu_dist/superlu_dist.c:720:25:
>>  error: 'LargeDiag_MC64' was not declared in this scope
>> 
>>  src/mat/impls/aij/mpi/superlu_dist/superlu_dist.c:723
>>
>> [http://ftp.mcs.anl.gov/pub/petsc/nightlylogs/archive/2018/06/12/build_next_arch-osx-10.6-cxx-cmplx-pkgs-dbg_ipro.log]
>>  
>> /Users/petsc/petsc.next-3/src/mat/impls/aij/mpi/superlu_dist/superlu_dist.c:723:25:
>>  error: use of undeclared identifier 'LargeDiag_AWPM'
>>
>> [http://ftp.mcs.anl.gov/pub/petsc/nightlylogs/archive/2018/06/12/build_next_arch-freebsd-cxx-pkgs-opt_wii.log]
>>  
>> /usr/home/balay/petsc.next-2/src/mat/impls/aij/mpi/superlu_dist/superlu_dist.c:723:25:
>>  error: 'LargeDiag_AWPM' was not declared in this scope
>>   

Re: [petsc-dev] superlu_dist: update to version v5.4.0

[Smith, Barry F.]

> On Jun 12, 2018, at 1:00 PM, Balay, Satish  wrote:
> 
> For now - removed from next [until I figureout how to handle it]
> 
> One motivation for this change is xsdk@devel build [wrt ATPESC tutorials].
> 
> Satish
> 
> On Tue, 12 Jun 2018, Smith, Barry F. wrote:
> 
>> 
>>  Complain to Sherry and have her fix it before moving the new release into 
>> PETSc
>> 
>>   Barry
>> 
>> 
>>> On Jun 12, 2018, at 10:59 AM, Satish Balay  wrote:
>>> 
>>> 
>>> $ diff superlu/SRC/superlu_enum_consts.h 
>>> superlu_dist/SRC/superlu_enum_consts.h
>>> 16a17
>>>> * January 28, 2018
>>> 28c29
>>> < typedef enum {NOROWPERM, LargeDiag, MY_PERMR}   rowperm_t;
>>> ---
>>>> typedef enum {NOROWPERM, LargeDiag_MC64, LargeDiag_AWPM, MY_PERMR} 
>>>> rowperm_t;
>>> 34c35,36
>>> < typedef enum {LUSUP, UCOL, LSUB, USUB, LLVL, ULVL}  MemType;
>>> ---
>>>> //typedef enum {LUSUP, UCOL, LSUB, USUB, LLVL, ULVL, NO_MEMTYPE}  MemType;
>>>> typedef enum {USUB, LSUB, UCOL, LUSUP, LLVL, ULVL, NO_MEMTYPE}  MemType;
>>> 69a72,74
>>>>   COMM_DIAG, /* Bcast diagonal block to process column */
>>>>   COMM_RIGHT, /* communicate L panel */
>>>>   COMM_DOWN, /* communicate U panel */
>>> 
>>> So superlu and superlu_dist share the same 'superlu_enum_consts.h'
>>> file - but with differences that are now incompatible with each other.
>>> 
>>> I'm not sure how to resolve this..
>>> 
>>> Satish
>>> 
>>> On Tue, 12 Jun 2018, PETSc checkBuilds wrote:
>>> 
>>>> 
>>>> 
>>>> Dear PETSc developer,
>>>> 
>>>> This email contains listings of contributions attributed to you by
>>>> `git blame` that caused compiler errors or warnings in PETSc automated
>>>> testing.  Follow the links to see the full log files. Please attempt to fix
>>>> the issues promptly or let us know at petsc-dev@mcs.anl.gov if you are 
>>>> unable
>>>> to resolve the issues.
>>>> 
>>>> Thanks,
>>>> The PETSc development team
>>>> 
>>>> 
>>>> 
>>>> warnings attributed to commit 
>>>> https://bitbucket.org/petsc/petsc/commits/8e05559
>>>> superlu_dist: update to version v5.4.0
>>>> 
>>>> src/mat/impls/aij/mpi/superlu_dist/superlu_dist.c:571
>>>>   
>>>> [http://ftp.mcs.anl.gov/pub/petsc/nightlylogs/archive/2018/06/12/build_next_arch-freebsd-cxx-pkgs-opt_wii.log]
>>>> 
>>>> /usr/home/balay/petsc.next-2/src/mat/impls/aij/mpi/superlu_dist/superlu_dist.c:571:8:
>>>>  error: 'LargeDiag_MC64' was not declared in this scope
>>>>   
>>>> [http://ftp.mcs.anl.gov/pub/petsc/nightlylogs/archive/2018/06/12/build_next_arch-linux-pkgs-cxx-mlib_el6.log]
>>>> 
>>>> /home/sandbox/petsc/petsc.next-3/src/mat/impls/aij/mpi/superlu_dist/superlu_dist.c:571:8:
>>>>  error: 'LargeDiag_MC64' was not declared in this scope
>>>>   
>>>> [http://ftp.mcs.anl.gov/pub/petsc/nightlylogs/archive/2018/06/12/build_next_arch-osx-10.6-cxx-cmplx-pkgs-dbg_ipro.log]
>>>> 
>>>> /Users/petsc/petsc.next-3/src/mat/impls/aij/mpi/superlu_dist/superlu_dist.c:571:8:
>>>>  error: use of undeclared identifier 'LargeDiag_MC64'
>>>> 
>>>> src/mat/impls/aij/mpi/superlu_dist/superlu_dist.c:574
>>>>   
>>>> [http://ftp.mcs.anl.gov/pub/petsc/nightlylogs/archive/2018/06/12/build_next_arch-freebsd-cxx-pkgs-opt_wii.log]
>>>> 
>>>> /usr/home/balay/petsc.next-2/src/mat/impls/aij/mpi/superlu_dist/superlu_dist.c:574:8:
>>>>  error: 'LargeDiag_AWPM' was not declared in this scope
>>>>   
>>>> [http://ftp.mcs.anl.gov/pub/petsc/nightlylogs/archive/2018/06/12/build_next_arch-linux-pkgs-cxx-mlib_el6.log]
>>>> 
>>>> /home/sandbox/petsc/petsc.next-3/src/mat/impls/aij/mpi/superlu_dist/superlu_dist.c:574:8:
>>>>  error: 'LargeDiag_AWPM' was not declared in this scope
>>>>   
>>>> [http://ftp.mcs.anl.gov/pub/petsc/nightlylogs/archive/2018/06/12/build_next_arch-osx-10.6-cxx-cmplx-pkgs-dbg_ipro.log]
>>>> 
>>>> /Users/petsc/petsc.next-3/src/mat/impls/aij/mpi/superlu_dist/superlu_dist.c:574:8:
>>>>  error: use of undeclared identifier 'Lar

Re: [petsc-dev] superlu_dist: update to version v5.4.0

[Smith, Barry F.]

> On Jun 12, 2018, at 9:36 PM, Satish Balay  wrote:
> 
> Sure - that will work.
> 
> But ultimately - if superlu and superlu_dist are separate packages
> (and meant to be used from the same applicaton)- they should not have
> any include files - that are common.
> 
> If they have common files - then there should be some version check
> that prevent the wrong version of the common file from used.
> 
> Or perhaps some other better organizaton of files between the packages
> is possible.
> 
> Alternative is - PETSc configure should eror out if one tries to use
> both packages at the same time.

   We don't like this at all. Applications should be able to switch between 
packages at runtime not require reconfiguring and rebuild etc.

Barry

> 
> Satish
> 
> On Tue, 12 Jun 2018, Xiaoye S. Li wrote:
> 
>> Satish,
>> 
>> I forgot to update sequential superlu. Right now I am busy with a paper
>> deadline on Friday. If I couldn't get to it before then, will certainly do
>> it on Saturday.
>> 
>> Is it sufficient to just update the master branch, without doing a new
>> tarball release?
>> 
>> Sherry
>> 
>> 
>> On Tue, Jun 12, 2018 at 11:00 AM, Satish Balay  wrote:
>> 
>>> For now - removed from next [until I figureout how to handle it]
>>> 
>>> One motivation for this change is xsdk@devel build [wrt ATPESC tutorials].
>>> 
>>> Satish
>>> 
>>> On Tue, 12 Jun 2018, Smith, Barry F. wrote:
>>> 
>>>> 
>>>>  Complain to Sherry and have her fix it before moving the new release
>>> into PETSc
>>>> 
>>>>   Barry
>>>> 
>>>> 
>>>>> On Jun 12, 2018, at 10:59 AM, Satish Balay  wrote:
>>>>> 
>>>>> 
>>>>> $ diff superlu/SRC/superlu_enum_consts.h
>>> superlu_dist/SRC/superlu_enum_consts.h
>>>>> 16a17
>>>>>> * January 28, 2018
>>>>> 28c29
>>>>> < typedef enum {NOROWPERM, LargeDiag, MY_PERMR}
>>> rowperm_t;
>>>>> ---
>>>>>> typedef enum {NOROWPERM, LargeDiag_MC64, LargeDiag_AWPM, MY_PERMR}
>>> rowperm_t;
>>>>> 34c35,36
>>>>> < typedef enum {LUSUP, UCOL, LSUB, USUB, LLVL, ULVL}
>>> MemType;
>>>>> ---
>>>>>> //typedef enum {LUSUP, UCOL, LSUB, USUB, LLVL, ULVL, NO_MEMTYPE}
>>> MemType;
>>>>>> typedef enum {USUB, LSUB, UCOL, LUSUP, LLVL, ULVL, NO_MEMTYPE}
>>> MemType;
>>>>> 69a72,74
>>>>>>   COMM_DIAG, /* Bcast diagonal block to process column */
>>>>>>   COMM_RIGHT, /* communicate L panel */
>>>>>>   COMM_DOWN, /* communicate U panel */
>>>>> 
>>>>> So superlu and superlu_dist share the same 'superlu_enum_consts.h'
>>>>> file - but with differences that are now incompatible with each other.
>>>>> 
>>>>> I'm not sure how to resolve this..
>>>>> 
>>>>> Satish
>>>>> 
>>>>> On Tue, 12 Jun 2018, PETSc checkBuilds wrote:
>>>>> 
>>>>>> 
>>>>>> 
>>>>>> Dear PETSc developer,
>>>>>> 
>>>>>> This email contains listings of contributions attributed to you by
>>>>>> `git blame` that caused compiler errors or warnings in PETSc automated
>>>>>> testing.  Follow the links to see the full log files. Please attempt
>>> to fix
>>>>>> the issues promptly or let us know at petsc-dev@mcs.anl.gov if you
>>> are unable
>>>>>> to resolve the issues.
>>>>>> 
>>>>>> Thanks,
>>>>>> The PETSc development team
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> warnings attributed to commit https://bitbucket.org/petsc/
>>> petsc/commits/8e05559
>>>>>> superlu_dist: update to version v5.4.0
>>>>>> 
>>>>>> src/mat/impls/aij/mpi/superlu_dist/superlu_dist.c:571
>>>>>>   [http://ftp.mcs.anl.gov/pub/petsc/nightlylogs/archive/
>>> 2018/06/12/build_next_arch-freebsd-cxx-pkgs-opt_wii.log]
>>>>>> 
>>>>>> /usr/home/balay/petsc.next-2/src/mat/impls/aij/mpi/superlu_dist/superlu_dist.c:571:8:
>>> error: 'LargeDiag_MC64' was not declared in this scope
>>>>>>   [http://ftp.mcs.anl.gov/pub/petsc/nightl