[petsc-dev] 32 bit compilers and PETSc

[Barry Smith]

   I just wasted an enormous amount of time "debugging"my additions to BAIJ on 
the solaris test machines. Turns out there was no bug at all just that the 
compilers default to 32 bit and we have no possible way of detecting size_t 
overflow; so smallish test cases can become a debugging nightmare. 

   Can we make ./configure ban 32 bit compilers unless a special flag is used? 
And just have one CI test that uses 32 bit where we turn off examples that 
overflow 32 bits?

  Barry

Re: [petsc-dev] [petsc-users] headsup: switch git default branch from 'master' to 'main'

[Satish Balay via petsc-dev]

An update: Looks like we have to do the following in our existing clones:

git remote set-head origin -a

Satish
-

balay@sb /home/balay/petsc (main=)
$ git gc
fatal: bad object refs/remotes/origin/HEAD
fatal: failed to run repack
balay@sb /home/balay/petsc (main=)
$ cat .git/refs/remotes/origin/HEAD
ref: refs/remotes/origin/master
balay@sb /home/balay/petsc (main=)
$ git remote set-head origin -a
origin/HEAD set to main
balay@sb /home/balay/petsc (main=)
$ cat .git/refs/remotes/origin/HEAD
ref: refs/remotes/origin/main
balay@sb /home/balay/petsc (main=)
$ git gc --prune=now
Enumerating objects: 942787, done.
Counting objects: 100% (942787/942787), done.
Delta compression using up to 4 threads
Compressing objects: 100% (213109/213109), done.
Writing objects: 100% (942787/942787), done.
Total 942787 (delta 725291), reused 942001 (delta 724664), pack-reused 0
Checking connectivity: 942787, done.
Expanding reachable commits in commit graph: 86669, done.
balay@sb /home/balay/petsc (main=)
$ 


On Fri, 26 Feb 2021, Satish Balay via petsc-users wrote:

> Update:
> 
> the switch (at gitlab.com/petsc/petsc) is done.
> 
> Please delete your local copy of 'master' branch and start using 'main' 
> branch.
> 
> Satish
> 
> On Tue, 23 Feb 2021, Satish Balay via petsc-dev wrote:
> 
> > All,
> > 
> > This is a heads-up, we are to switch the default branch in petsc git
> > repo from 'master' to 'main'
> > 
> > [Will plan to do the switch on friday the 26th]
> > 
> > We've previously switched 'maint' branch to 'release' before 3.14
> > release - and this change (to 'main') is the next step in this direction.
> > 
> > Satish
> > 
> 

Re: [petsc-dev] Petsc "make test" have more failures for --with-openmp=1

[Eric Chamberland]

Hi,

Just to add something I didn't knew before: I got an answer from intel 
and tried the mkl_link_tool script shipped with mkl (into OneAPI package):


$MKLROOT/bin/intel64/mkl_link_tool -libs --compiler=gnu_c --arch=intel64 
--linking=dynamic --parallel=yes --interface=lp64 
--threading-library=gomp --mpi=openmpi --cluster_library=scalapack



    Intel(R) oneAPI Math Kernel Library (oneMKL) Link Tool v6.1
   ==

Output
==

Linking line:
 -L${MKLROOT}/lib/intel64 -lmkl_scalapack_lp64 -Wl,--no-as-needed 
-lmkl_intel_lp64 -lmkl_gnu_thread -lmkl_core -lmkl_blacs_openmpi_lp64 
-lgomp -lpthread -lm -ldl


I don't know if this is intersting for PETSc configuration scripts to 
try to use this or not, but it exists!


(see 
https://community.intel.com/t5/Intel-oneAPI-Math-Kernel-Library/Intel-MKL-Link-Line-Advisor-as-external-tool/m-p/1260895#M30974 
)


Eric


On 2021-03-03 11:13 a.m., Barry Smith wrote:


2) I can reproduce the src/mat/tests/ex242.c error (which explicitly 
uses ScaLAPACK, none of the above PC uses it explicitly, except 
PCBDDC/PCHPDDM when using MUMPS on “big” problems where root nodes 
are factorized using ScaLAPACK, see -mat_mumps_icntl_13)
3) I’m seeing that both on your machine and mine, PETSc BuildSystem 
insist on linking libmkl_blacs_intelmpi_lp64.so even though we 
supply explicitly libmkl_blacs_openmpi_lp64.so

This for example yields a wrong Makefile.inc for MUMPS:
$ cat 
arch-linux2-c-opt-ompi/externalpackages/MUMPS_5.3.5/Makefile.inc|grep 
blacs

SCALAP  = […] -lmkl_blacs_openmpi_lp64
LIBBLAS = […] -lmkl_blacs_intelmpi_lp64 -lgomp -ldl -lpthread -lm […]

Despite what Barry says, I think PETSc is partially to blame as well 
(why use libmkl_blacs_intelmpi_lp64.so even though BuildSystem is 
capable of detecting we are using OpenMPI).

I’ll try to fix this to see if it solves 2).


Okay, that's a very nice finding!!!  Hope it will be "fixable" easily!


The knowledge is there but the information may not be trivially 
available to make the right decisions. Parts of the BLAS/LAPACK checks 
use the "check everything" approach. For example


# Look for Multi-Threaded MKL for MKL_C/Pardiso
      useCPardiso=0
      usePardiso=0
      if self.argDB['with-mkl_cpardiso'] or 'with-mkl_cpardiso-dir' in 
self.argDB or 'with-mkl_cpardiso-lib' in self.argDB:

        useCPardiso=1
mkl_blacs_64=[['mkl_blacs_intelmpi'+ILP64+''],['mkl_blacs_mpich'+ILP64+''],['mkl_blacs_sgimpt'+ILP64+''],['mkl_blacs_openmpi'+ILP64+'']]
mkl_blacs_32=[['mkl_blacs_intelmpi'],['mkl_blacs_mpich'],['mkl_blacs_sgimpt'],['mkl_blacs_openmpi']]
      elif self.argDB['with-mkl_pardiso'] or 'with-mkl_pardiso-dir' in 
self.argDB or 'with-mkl_pardiso-lib' in self.argDB:

        usePardiso=1
        mkl_blacs_64=[[]]
        mkl_blacs_32=[[]]
      if useCPardiso or usePardiso:
        self.logPrintBox('BLASLAPACK: Looking for Multithreaded MKL 
for C/Pardiso')
        for libdir in 
[os.path.join('lib','64'),os.path.join('lib','ia64'),os.path.join('lib','em64t'),os.path.join('lib','intel64'),'lib','64','ia64','em64t','intel64',

 os.path.join('lib','32'),os.path.join('lib','ia32'),'32','ia32','']:
          if not os.path.exists(os.path.join(dir,libdir)):
            self.logPrint('MKL Path not found.. skipping: 
'+os.path.join(dir,libdir))

          else:
            self.log.write('Files and directories in that 
directory:\n'+str(os.listdir(os.path.join(dir,libdir)))+'\n')
            #  iomp5 is provided by the Intel compilers on MacOS. Run 
source /opt/intel/bin/compilervars.sh intel64 to have it added to 
LIBRARY_PATH
            #  then locate libimp5.dylib in the LIBRARY_PATH and copy 
it to os.path.join(dir,libdir)

            for i in mkl_blacs_64:
              yield ('User specified MKL-C/Pardiso Intel-Linux64', 
None, 
[os.path.join(dir,libdir,'libmkl_intel'+ILP64+'.a'),'mkl_core','mkl_intel_thread']+i+['iomp5','dl','pthread'],known,'yes')
              yield ('User specified MKL-C/Pardiso GNU-Linux64', None, 
[os.path.join(dir,libdir,'libmkl_intel'+ILP64+'.a'),'mkl_core','mkl_gnu_thread']+i+['gomp','dl','pthread'],known,'yes')
              yield ('User specified MKL-Pardiso Intel-Windows64', 
None, 
[os.path.join(dir,libdir,'mkl_core.lib'),'mkl_intel'+ILP64+'.lib','mkl_intel_thread.lib']+i+['libiomp5md.lib'],known,'yes')

            for i in mkl_blacs_32:
              yield ('User specified MKL-C/Pardiso Intel-Linux32', 
None, 
[os.path.join(dir,libdir,'libmkl_intel.a'),'mkl_core','mkl_intel_thread']+i+['iomp5','dl','pthread'],'32','yes')
              yield ('User specified MKL-C/Pardiso GNU-Linux32', None, 
[os.path.join(dir,libdir,'libmkl_intel.a'),'mkl_core','mkl_gnu_thread']+i+['gomp','dl','pthread'],'32','yes')
              yield ('User specified MKL-Pardiso Intel-Windows32', 
None, 
[os.path.join(dir,libdir,'mkl_core.lib'),'mkl_intel_c.lib','mkl_intel_thread.lib']+i+['libiomp5md.lib'],'32','yes')

        return

The assumption is that 

Re: [petsc-dev] Commit squashing in MR

[Jacob Faibussowitsch]

This is certainly useful, prior to this I was using 

git diff $(git merge-base --fork-point main)

To see branch changes (but this would dump all changes in all files in 
terminal, so quite unwieldy for large diffs)...

Best regards,

Jacob Faibussowitsch
(Jacob Fai - booss - oh - vitch)
Cell: (312) 694-3391

> On Mar 3, 2021, at 14:12, Satish Balay  wrote:
> 
> Also: sometimes I don't want to see the individual commits - just the overall 
> change in this branch [aka the MR 'changes' view]
> 
> git diff main...origin/jacobf/2020-09-09/feature/petscmapping/future
> 
> Only diff for include/petsc.h
> 
> git diff main...origin/jacobf/2020-09-09/feature/petscmapping/future 
> include/petsc.h
> 
> Satish
> 
> 
> On Wed, 3 Mar 2021, Satish Balay via petsc-dev wrote:
> 
>> And I frequently use it - for ex, to check (only) the commits in a branch - 
>> say origin/jacobf/2020-09-09/feature/petscmapping/future
>> 
>> gitk main..origin/jacobf/2020-09-09/feature/petscmapping/future
>> 
>> [equivalent to "git log 
>> main..origin/jacobf/2020-09-09/feature/petscmapping/future"]
>> 
>> And then check the diffs for individual commit (as needed) via this interface
>> 
>> Check only the diffs for changes to include/petsc.h in this branch
>> 
>> gitk main..origin/jacobf/2020-09-09/feature/petscmapping/future 
>> include/petsc.h
>> 
>> Satish
>> 
>> On Wed, 3 Mar 2021, Jacob Faibussowitsch wrote:
>> 
 'gitk' is easier to read [for me] than 'git log --graph'
>>> 
>>> Where was this my entire life… best kept git secret!
>>> 
>>> Best regards,
>>> 
>>> Jacob Faibussowitsch
>>> (Jacob Fai - booss - oh - vitch)
>>> Cell: (312) 694-3391
>>> 
 On Mar 3, 2021, at 13:55, Satish Balay  wrote:
 
 'gitk' is easier to read [for me] than 'git log --graph'
 
 Satish
 
 On Wed, 3 Mar 2021, Jacob Faibussowitsch wrote:
 
>> git: 'graph' is not a git command. See 'git --help'.
> 
> I have it as an alias:
> 
> graph = !git log --graph --pretty=format:'%Cred%h%Creset 
> -%C(yellow)%d%Creset %s %Cgreen(%cr) %C(bold blue)<%an>%Creset' 
> --abbrev-commit --date=relative
> 
> Best regards,
> 
> Jacob Faibussowitsch
> (Jacob Fai - booss - oh - vitch)
> Cell: (312) 694-3391
> 
>> On Mar 3, 2021, at 13:50, Mark Adams  wrote:
>> 
>> 
>> 
>> On Tue, Mar 2, 2021 at 10:02 PM Junchao Zhang > > wrote:
>> I am a naive git user, so I use interactive git rebase.  Suppose I am on 
>> the branch I want to modify, 
>> 
>> 1) Use git graph to locate an upstream commit to be used as the base
>> $ git graph
>> 
>> Humm 
>> 
>> 14:49 adams/cusparse-lu-landau= /gpfs/alpine/csc314/scratch/adams/petsc$ 
>> git --version
>> git version 2.20.1
>> 14:49 adams/cusparse-lu-landau= /gpfs/alpine/csc314/scratch/adams/petsc$ 
>> git graph
>> git: 'graph' is not a git command. See 'git --help'.
>> 
>> The most similar commands are
>> branch
>> grep
>> 
> 
> 
 
>>> 
>>> 
>> 

Re: [petsc-dev] Commit squashing in MR

[Satish Balay via petsc-dev]

Also: sometimes I don't want to see the individual commits - just the overall 
change in this branch [aka the MR 'changes' view]

git diff main...origin/jacobf/2020-09-09/feature/petscmapping/future

Only diff for include/petsc.h

git diff main...origin/jacobf/2020-09-09/feature/petscmapping/future 
include/petsc.h

Satish


On Wed, 3 Mar 2021, Satish Balay via petsc-dev wrote:

> And I frequently use it - for ex, to check (only) the commits in a branch - 
> say origin/jacobf/2020-09-09/feature/petscmapping/future
> 
> gitk main..origin/jacobf/2020-09-09/feature/petscmapping/future
> 
> [equivalent to "git log 
> main..origin/jacobf/2020-09-09/feature/petscmapping/future"]
> 
> And then check the diffs for individual commit (as needed) via this interface
> 
> Check only the diffs for changes to include/petsc.h in this branch
> 
> gitk main..origin/jacobf/2020-09-09/feature/petscmapping/future 
> include/petsc.h
> 
> Satish
> 
> On Wed, 3 Mar 2021, Jacob Faibussowitsch wrote:
> 
> > > 'gitk' is easier to read [for me] than 'git log --graph'
> > 
> > Where was this my entire life… best kept git secret!
> > 
> > Best regards,
> > 
> > Jacob Faibussowitsch
> > (Jacob Fai - booss - oh - vitch)
> > Cell: (312) 694-3391
> > 
> > > On Mar 3, 2021, at 13:55, Satish Balay  wrote:
> > > 
> > > 'gitk' is easier to read [for me] than 'git log --graph'
> > > 
> > > Satish
> > > 
> > > On Wed, 3 Mar 2021, Jacob Faibussowitsch wrote:
> > > 
> > >>> git: 'graph' is not a git command. See 'git --help'.
> > >> 
> > >> I have it as an alias:
> > >> 
> > >> graph = !git log --graph --pretty=format:'%Cred%h%Creset 
> > >> -%C(yellow)%d%Creset %s %Cgreen(%cr) %C(bold blue)<%an>%Creset' 
> > >> --abbrev-commit --date=relative
> > >> 
> > >> Best regards,
> > >> 
> > >> Jacob Faibussowitsch
> > >> (Jacob Fai - booss - oh - vitch)
> > >> Cell: (312) 694-3391
> > >> 
> > >>> On Mar 3, 2021, at 13:50, Mark Adams  wrote:
> > >>> 
> > >>> 
> > >>> 
> > >>> On Tue, Mar 2, 2021 at 10:02 PM Junchao Zhang  > >>> > wrote:
> > >>> I am a naive git user, so I use interactive git rebase.  Suppose I am 
> > >>> on the branch I want to modify, 
> > >>> 
> > >>> 1) Use git graph to locate an upstream commit to be used as the base
> > >>> $ git graph
> > >>> 
> > >>> Humm 
> > >>> 
> > >>> 14:49 adams/cusparse-lu-landau= 
> > >>> /gpfs/alpine/csc314/scratch/adams/petsc$ git --version
> > >>> git version 2.20.1
> > >>> 14:49 adams/cusparse-lu-landau= 
> > >>> /gpfs/alpine/csc314/scratch/adams/petsc$ git graph
> > >>> git: 'graph' is not a git command. See 'git --help'.
> > >>> 
> > >>> The most similar commands are
> > >>> branch
> > >>> grep
> > >>> 
> > >> 
> > >> 
> > > 
> > 
> > 
> 

Re: [petsc-dev] Commit squashing in MR

[Satish Balay via petsc-dev]

And I frequently use it - for ex, to check (only) the commits in a branch - say 
origin/jacobf/2020-09-09/feature/petscmapping/future

gitk main..origin/jacobf/2020-09-09/feature/petscmapping/future

[equivalent to "git log 
main..origin/jacobf/2020-09-09/feature/petscmapping/future"]

And then check the diffs for individual commit (as needed) via this interface

Check only the diffs for changes to include/petsc.h in this branch

gitk main..origin/jacobf/2020-09-09/feature/petscmapping/future include/petsc.h

Satish

On Wed, 3 Mar 2021, Jacob Faibussowitsch wrote:

> > 'gitk' is easier to read [for me] than 'git log --graph'
> 
> Where was this my entire life… best kept git secret!
> 
> Best regards,
> 
> Jacob Faibussowitsch
> (Jacob Fai - booss - oh - vitch)
> Cell: (312) 694-3391
> 
> > On Mar 3, 2021, at 13:55, Satish Balay  wrote:
> > 
> > 'gitk' is easier to read [for me] than 'git log --graph'
> > 
> > Satish
> > 
> > On Wed, 3 Mar 2021, Jacob Faibussowitsch wrote:
> > 
> >>> git: 'graph' is not a git command. See 'git --help'.
> >> 
> >> I have it as an alias:
> >> 
> >> graph = !git log --graph --pretty=format:'%Cred%h%Creset 
> >> -%C(yellow)%d%Creset %s %Cgreen(%cr) %C(bold blue)<%an>%Creset' 
> >> --abbrev-commit --date=relative
> >> 
> >> Best regards,
> >> 
> >> Jacob Faibussowitsch
> >> (Jacob Fai - booss - oh - vitch)
> >> Cell: (312) 694-3391
> >> 
> >>> On Mar 3, 2021, at 13:50, Mark Adams  wrote:
> >>> 
> >>> 
> >>> 
> >>> On Tue, Mar 2, 2021 at 10:02 PM Junchao Zhang  >>> > wrote:
> >>> I am a naive git user, so I use interactive git rebase.  Suppose I am on 
> >>> the branch I want to modify, 
> >>> 
> >>> 1) Use git graph to locate an upstream commit to be used as the base
> >>> $ git graph
> >>> 
> >>> Humm 
> >>> 
> >>> 14:49 adams/cusparse-lu-landau= /gpfs/alpine/csc314/scratch/adams/petsc$ 
> >>> git --version
> >>> git version 2.20.1
> >>> 14:49 adams/cusparse-lu-landau= /gpfs/alpine/csc314/scratch/adams/petsc$ 
> >>> git graph
> >>> git: 'graph' is not a git command. See 'git --help'.
> >>> 
> >>> The most similar commands are
> >>> branch
> >>> grep
> >>> 
> >> 
> >> 
> > 
> 
> 

Re: [petsc-dev] Commit squashing in MR

[Jacob Faibussowitsch]

> 'gitk' is easier to read [for me] than 'git log --graph'

Where was this my entire life… best kept git secret!

Best regards,

Jacob Faibussowitsch
(Jacob Fai - booss - oh - vitch)
Cell: (312) 694-3391

> On Mar 3, 2021, at 13:55, Satish Balay  wrote:
> 
> 'gitk' is easier to read [for me] than 'git log --graph'
> 
> Satish
> 
> On Wed, 3 Mar 2021, Jacob Faibussowitsch wrote:
> 
>>> git: 'graph' is not a git command. See 'git --help'.
>> 
>> I have it as an alias:
>> 
>> graph = !git log --graph --pretty=format:'%Cred%h%Creset 
>> -%C(yellow)%d%Creset %s %Cgreen(%cr) %C(bold blue)<%an>%Creset' 
>> --abbrev-commit --date=relative
>> 
>> Best regards,
>> 
>> Jacob Faibussowitsch
>> (Jacob Fai - booss - oh - vitch)
>> Cell: (312) 694-3391
>> 
>>> On Mar 3, 2021, at 13:50, Mark Adams  wrote:
>>> 
>>> 
>>> 
>>> On Tue, Mar 2, 2021 at 10:02 PM Junchao Zhang >> > wrote:
>>> I am a naive git user, so I use interactive git rebase.  Suppose I am on 
>>> the branch I want to modify, 
>>> 
>>> 1) Use git graph to locate an upstream commit to be used as the base
>>> $ git graph
>>> 
>>> Humm 
>>> 
>>> 14:49 adams/cusparse-lu-landau= /gpfs/alpine/csc314/scratch/adams/petsc$ 
>>> git --version
>>> git version 2.20.1
>>> 14:49 adams/cusparse-lu-landau= /gpfs/alpine/csc314/scratch/adams/petsc$ 
>>> git graph
>>> git: 'graph' is not a git command. See 'git --help'.
>>> 
>>> The most similar commands are
>>> branch
>>> grep
>>> 
>> 
>> 
> 

Re: [petsc-dev] Commit squashing in MR

[Mark Adams]

On Wed, Mar 3, 2021 at 2:53 PM Jacob Faibussowitsch 
wrote:

> git: 'graph' is not a git command. See 'git --help'.
>
>
> I have it as an alias:
>
> graph = !git log --graph --pretty=format:'%Cred%h%Creset
> -%C(yellow)%d%Creset %s %Cgreen(%cr) %C(bold blue)<%an>%Creset'
> --abbrev-commit --date=relative
>

Ah, I like that. I don't have gitk on all my machines.
Thanks,


>
> Best regards,
>
> Jacob Faibussowitsch
> (Jacob Fai - booss - oh - vitch)
> Cell: (312) 694-3391
>
> On Mar 3, 2021, at 13:50, Mark Adams  wrote:
>
>
>
> On Tue, Mar 2, 2021 at 10:02 PM Junchao Zhang 
> wrote:
>
>> I am a naive git user, so I use interactive git rebase.  Suppose I am on
>> the branch I want to modify,
>>
>> 1) Use git graph to locate an upstream commit to be used as the base
>> $ git graph
>>
>
> Humm 
>
> 14:49 adams/cusparse-lu-landau= /gpfs/alpine/csc314/scratch/adams/petsc$
> git --version
> git version 2.20.1
> 14:49 adams/cusparse-lu-landau= /gpfs/alpine/csc314/scratch/adams/petsc$
> git graph
> git: 'graph' is not a git command. See 'git --help'.
>
> The most similar commands are
> branch
> grep
>
>
>
>

Re: [petsc-dev] Commit squashing in MR

[Junchao Zhang]

Oh, graph is an alias in my .gitconfig

[alias]
graph = log --graph --decorate --abbrev-commit --pretty=oneline

--Junchao Zhang


On Wed, Mar 3, 2021 at 1:51 PM Mark Adams  wrote:

>
>
> On Tue, Mar 2, 2021 at 10:02 PM Junchao Zhang 
> wrote:
>
>> I am a naive git user, so I use interactive git rebase.  Suppose I am on
>> the branch I want to modify,
>>
>> 1) Use git graph to locate an upstream commit to be used as the base
>> $ git graph
>>
>
> Humm 
>
> 14:49 adams/cusparse-lu-landau= /gpfs/alpine/csc314/scratch/adams/petsc$
> git --version
> git version 2.20.1
> 14:49 adams/cusparse-lu-landau= /gpfs/alpine/csc314/scratch/adams/petsc$
> git graph
> git: 'graph' is not a git command. See 'git --help'.
>
> The most similar commands are
> branch
> grep
>
>

Re: [petsc-dev] Commit squashing in MR

[Satish Balay via petsc-dev]

'gitk' is easier to read [for me] than 'git log --graph'

Satish

On Wed, 3 Mar 2021, Jacob Faibussowitsch wrote:

> > git: 'graph' is not a git command. See 'git --help'.
> 
> I have it as an alias:
> 
> graph = !git log --graph --pretty=format:'%Cred%h%Creset -%C(yellow)%d%Creset 
> %s %Cgreen(%cr) %C(bold blue)<%an>%Creset' --abbrev-commit --date=relative
> 
> Best regards,
> 
> Jacob Faibussowitsch
> (Jacob Fai - booss - oh - vitch)
> Cell: (312) 694-3391
> 
> > On Mar 3, 2021, at 13:50, Mark Adams  wrote:
> > 
> > 
> > 
> > On Tue, Mar 2, 2021 at 10:02 PM Junchao Zhang  > > wrote:
> > I am a naive git user, so I use interactive git rebase.  Suppose I am on 
> > the branch I want to modify, 
> > 
> > 1) Use git graph to locate an upstream commit to be used as the base
> > $ git graph
> > 
> > Humm 
> > 
> > 14:49 adams/cusparse-lu-landau= /gpfs/alpine/csc314/scratch/adams/petsc$ 
> > git --version
> > git version 2.20.1
> > 14:49 adams/cusparse-lu-landau= /gpfs/alpine/csc314/scratch/adams/petsc$ 
> > git graph
> > git: 'graph' is not a git command. See 'git --help'.
> > 
> > The most similar commands are
> > branch
> > grep
> >  
> 
> 

Re: [petsc-dev] Commit squashing in MR

[Jacob Faibussowitsch]

> git: 'graph' is not a git command. See 'git --help'.

I have it as an alias:

graph = !git log --graph --pretty=format:'%Cred%h%Creset -%C(yellow)%d%Creset 
%s %Cgreen(%cr) %C(bold blue)<%an>%Creset' --abbrev-commit --date=relative

Best regards,

Jacob Faibussowitsch
(Jacob Fai - booss - oh - vitch)
Cell: (312) 694-3391

> On Mar 3, 2021, at 13:50, Mark Adams  wrote:
> 
> 
> 
> On Tue, Mar 2, 2021 at 10:02 PM Junchao Zhang  > wrote:
> I am a naive git user, so I use interactive git rebase.  Suppose I am on the 
> branch I want to modify, 
> 
> 1) Use git graph to locate an upstream commit to be used as the base
> $ git graph
> 
> Humm 
> 
> 14:49 adams/cusparse-lu-landau= /gpfs/alpine/csc314/scratch/adams/petsc$ git 
> --version
> git version 2.20.1
> 14:49 adams/cusparse-lu-landau= /gpfs/alpine/csc314/scratch/adams/petsc$ git 
> graph
> git: 'graph' is not a git command. See 'git --help'.
> 
> The most similar commands are
> branch
> grep
>  

Re: [petsc-dev] Commit squashing in MR

[Satish Balay via petsc-dev]

On Wed, 3 Mar 2021, Mark Adams wrote:

> On Tue, Mar 2, 2021 at 11:09 PM Jacob Faibussowitsch 
> wrote:
> 
> >   I do not get this. I thought that rebasing with main put all the main
> > changes in before your commits. I have never seen any interspersed, so I do
> > not understand this.
> >
> >
> > Mark, do you rebase your branch over main first or do you merge main intro
> > your branch to update it?
> >
> 
> I never merge anymore. just rebase over main.
> 
> 
> > If you rebase, you can pick an N such that you only get your commits from
> > the branch.
> >
> 
> Yea, but what is N? If you know that then use it. I just guess if I lose
> track (I bounce between my Mac and two GPU machines so I can do a lot of
> wip commits) and as long as you guess too big, you will see where your
> commits start and ignore the rest.

If making simultaneous changes over multiple clones (to the same branch) - 'git 
pull --rebase' can be useful.

Satish

> 
> 
> >
> > Best regards,
> >
> > Jacob Faibussowitsch
> > (Jacob Fai - booss - oh - vitch)
> > Cell: (312) 694-3391
> >
> > On Mar 2, 2021, at 22:02, Barry Smith  wrote:
> >
> >
> >
> > On Mar 2, 2021, at 9:24 PM, Mark Adams  wrote:
> >
> > Ah, 'git graph' I will try that next time.
> >
> > I use 'git rebase -i HEAD~N', but you need an N.
> >
> > After you 'git rebase origin/main' you get other commits interspersed in
> > with yours,
> >
> >
> >I do not get this. I thought that rebasing with main put all the main
> > changes in before your commits. I have never seen any interspersed, so I do
> > not understand this.
> >
> >   Barry
> >
> > so I try to rebase -i before rebasing over main. Then rebase over main and
> > you have a clean and updated branch.
> >
> > Pick N to be large enough to cover the commits that you want to clean up.
> > Don't touch the ones that are not yours from main, the last time you
> > rebased over main.
> >
> > On Tue, Mar 2, 2021 at 10:02 PM Junchao Zhang 
> > wrote:
> >
> >> I am a naive git user, so I use interactive git rebase.  Suppose I am on
> >> the branch I want to modify,
> >>
> >> 1) Use git graph to locate an upstream commit to be used as the base
> >> $ git graph
> >> * 0d5433e9 (HEAD -> jczhang/sf-change-api) SF: rename SFCreateEmbeddedSF
> >> to SFCreateEmbeddedRootSF
> >> * e7314fbb SF: add an MPI_Op argument to SFBcast
> >> * 83df288d Replace MPIU_REPLACE with MPI_REPLACE
> >> *   b434c516 Merge branch
> >> 'barry/2021-02-02/petscsf-communication-specific' into 'main'
> >> |\
> >> | * 62152ded (barry/2021-02-02/petscsf-communication-specific)
> >> PetscSFView() never called viewer for the specific type (bug), hence many
> >> output files were incorrect.
> >> * |   a4f5d9b4 Merge branch 'jose/upgrade-magma' into 'main'
> >>
> >> 2) Suppose we choose b434c516 as the base. All commits we want to squash
> >> are after it.  Do interactive git rebase. It shows a screen for you to
> >> edit.  Read the help, which is helpful for new users
> >>   $ git rebase -i b434c516
> >> pick 83df288d Replace MPIU_REPLACE with MPI_REPLACE
> >> pick e7314fbb SF: add an MPI_Op argument to SFBcast
> >> pick 0d5433e9 SF: rename SFCreateEmbeddedSF to SFCreateEmbeddedRootSF
> >>
> >> # Rebase b434c516..0d5433e9 onto b434c516 (3 commands)
> >> #
> >> # Commands:
> >> # p, pick  = use commit
> >> # r, reword  = use commit, but edit the commit message
> >> # e, edit  = use commit, but stop for amending
> >> # s, squash  = use commit, but meld into previous commit
> >> # f, fixup  = like "squash", but discard this commit's log message
> >> # x, exec  = run command (the rest of the line) using shell
> >> # b, break = stop here (continue rebase later with 'git rebase
> >> --continue')
> >> # d, drop  = remove commit
> >> # l, label  = label current HEAD with a name
> >> # t, reset  = reset HEAD to a label
> >> # m, merge [-C  | -c ]  [# ]
> >> # .   create a merge commit using the original merge commit's
> >> # .   message (or the oneline, if no original merge commit was
> >> # .   specified). Use -c  to reword the commit message.
> >> #
> >> # These lines can be re-ordered; they are executed from top to bottom.
> >> #
> >> # If you remove a line here THAT COMMIT WILL BE LOST.
> >> #
> >> # However, if you remove everything, the rebase will be aborted.
> >> #
> >> # Note that empty commits are commented out
> >>
> >> 3) Suppose we want to squash the last two commits to 83df288d, replace
> >> their pick with s (or f, see the help for difference), save and exit the
> >> screen
> >> pick 83df288d Replace MPIU_REPLACE with MPI_REPLACE
> >> s e7314fbb SF: add an MPI_Op argument to SFBcast
> >> s 0d5433e9 SF: rename SFCreateEmbeddedSF to SFCreateEmbeddedRootSF
> >>
> >> A new screen shows up
> >>
> >> # This is a combination of 3 commits.
> >> # This is the 1st commit message:
> >>
> >> Replace MPIU_REPLACE with MPI_REPLACE
> >>
> >> Since we believe all MPI implementations support MPI_REPLACE
> >>
> >> # This is the commit message #2:
> >>
> >> SF: add 

Re: [petsc-dev] Commit squashing in MR

[Mark Adams]

On Tue, Mar 2, 2021 at 10:02 PM Junchao Zhang 
wrote:

> I am a naive git user, so I use interactive git rebase.  Suppose I am on
> the branch I want to modify,
>
> 1) Use git graph to locate an upstream commit to be used as the base
> $ git graph
>

Humm 

14:49 adams/cusparse-lu-landau= /gpfs/alpine/csc314/scratch/adams/petsc$
git --version
git version 2.20.1
14:49 adams/cusparse-lu-landau= /gpfs/alpine/csc314/scratch/adams/petsc$
git graph
git: 'graph' is not a git command. See 'git --help'.

The most similar commands are
branch
grep

Re: [petsc-dev] Commit squashing in MR

[Satish Balay via petsc-dev]

On Wed, 3 Mar 2021, Mark Adams wrote:

> On Tue, Mar 2, 2021 at 11:02 PM Barry Smith  wrote:
> 
> >
> >
> > On Mar 2, 2021, at 9:24 PM, Mark Adams  wrote:
> >
> > Ah, 'git graph' I will try that next time.
> >
> > I use 'git rebase -i HEAD~N', but you need an N.
> >
> > After you 'git rebase origin/main' you get other commits interspersed in
> > with yours,
> >
> >
> >I do not get this. I thought that rebasing with main put all the main
> > changes in before your commits. I have never seen any interspersed, so I do
> > not understand this.
> >
> 
> Humm, I'm not sure. The important thing is that you will hit main commits
> and you don't want to touch those or the first of your commits.

Best to think of 'squash (rebase -i (squash))' as different from 'rebase over 
branch'

Sure you can club both operations - assuming that's the intention.

In some cases its preferable to only do the squash - and not rebase
(over main). For ex: when you successfully run a test pipeline - best
to do 'squash-only' (after testing) to keep the test results current.

Satish

> 
> 
> >
> >   Barry
> >
> > so I try to rebase -i before rebasing over main. Then rebase over main and
> > you have a clean and updated branch.
> >
> > Pick N to be large enough to cover the commits that you want to clean up.
> > Don't touch the ones that are not yours from main, the last time you
> > rebased over main.
> >
> > On Tue, Mar 2, 2021 at 10:02 PM Junchao Zhang 
> > wrote:
> >
> >> I am a naive git user, so I use interactive git rebase.  Suppose I am on
> >> the branch I want to modify,
> >>
> >> 1) Use git graph to locate an upstream commit to be used as the base
> >> $ git graph
> >> * 0d5433e9 (HEAD -> jczhang/sf-change-api) SF: rename SFCreateEmbeddedSF
> >> to SFCreateEmbeddedRootSF
> >> * e7314fbb SF: add an MPI_Op argument to SFBcast
> >> * 83df288d Replace MPIU_REPLACE with MPI_REPLACE
> >> *   b434c516 Merge branch
> >> 'barry/2021-02-02/petscsf-communication-specific' into 'main'
> >> |\
> >> | * 62152ded (barry/2021-02-02/petscsf-communication-specific)
> >> PetscSFView() never called viewer for the specific type (bug), hence many
> >> output files were incorrect.
> >> * |   a4f5d9b4 Merge branch 'jose/upgrade-magma' into 'main'
> >>
> >> 2) Suppose we choose b434c516 as the base. All commits we want to squash
> >> are after it.  Do interactive git rebase. It shows a screen for you to
> >> edit.  Read the help, which is helpful for new users
> >>   $ git rebase -i b434c516
> >> pick 83df288d Replace MPIU_REPLACE with MPI_REPLACE
> >> pick e7314fbb SF: add an MPI_Op argument to SFBcast
> >> pick 0d5433e9 SF: rename SFCreateEmbeddedSF to SFCreateEmbeddedRootSF
> >>
> >> # Rebase b434c516..0d5433e9 onto b434c516 (3 commands)
> >> #
> >> # Commands:
> >> # p, pick  = use commit
> >> # r, reword  = use commit, but edit the commit message
> >> # e, edit  = use commit, but stop for amending
> >> # s, squash  = use commit, but meld into previous commit
> >> # f, fixup  = like "squash", but discard this commit's log message
> >> # x, exec  = run command (the rest of the line) using shell
> >> # b, break = stop here (continue rebase later with 'git rebase
> >> --continue')
> >> # d, drop  = remove commit
> >> # l, label  = label current HEAD with a name
> >> # t, reset  = reset HEAD to a label
> >> # m, merge [-C  | -c ]  [# ]
> >> # .   create a merge commit using the original merge commit's
> >> # .   message (or the oneline, if no original merge commit was
> >> # .   specified). Use -c  to reword the commit message.
> >> #
> >> # These lines can be re-ordered; they are executed from top to bottom.
> >> #
> >> # If you remove a line here THAT COMMIT WILL BE LOST.
> >> #
> >> # However, if you remove everything, the rebase will be aborted.
> >> #
> >> # Note that empty commits are commented out
> >>
> >> 3) Suppose we want to squash the last two commits to 83df288d, replace
> >> their pick with s (or f, see the help for difference), save and exit the
> >> screen
> >> pick 83df288d Replace MPIU_REPLACE with MPI_REPLACE
> >> s e7314fbb SF: add an MPI_Op argument to SFBcast
> >> s 0d5433e9 SF: rename SFCreateEmbeddedSF to SFCreateEmbeddedRootSF
> >>
> >> A new screen shows up
> >>
> >> # This is a combination of 3 commits.
> >> # This is the 1st commit message:
> >>
> >> Replace MPIU_REPLACE with MPI_REPLACE
> >>
> >> Since we believe all MPI implementations support MPI_REPLACE
> >>
> >> # This is the commit message #2:
> >>
> >> SF: add an MPI_Op argument to SFBcast
> >>
> >> # This is the commit message #3:
> >>
> >> SF: rename SFCreateEmbeddedSF to SFCreateEmbeddedRootSF
> >>
> >> # Please enter the commit message for your changes. Lines starting
> >> # with '#' will be ignored, and an empty message aborts the commit.
> >>
> >> 4) Edit the commit message as you want, save and exit, done!
> >>
> >> --Junchao Zhang
> >>
> >>
> >> On Tue, Mar 2, 2021 at 6:19 PM Blaise A Bourdin  wrote:
> >>
> >>> Hi,
> >>>
> >>> This is not 

Re: [petsc-dev] Commit squashing in MR

[Mark Adams]

On Tue, Mar 2, 2021 at 11:33 PM Jed Brown  wrote:

> Satish Balay via petsc-dev  writes:
>
> > On Wed, 3 Mar 2021, Blaise A Bourdin wrote:
> >
> >> Hi,
> >>
> >> This is not technically a petsc question.
> >> It would be great to have a short section in the PETSc integration
> workflow document explaining how to squash commits in a MR for git-impaired
> developers like me.
> >>
> >> Anybody wants to pitch in, or explain me how to do this?
> >
> > To squash commits - I use the 'squash' action in 'git rebase -i HASH'
> and figure out the HASH to use from 'gitk main..branch'
> >
> > [as git rebase requires the commit prior to the first commit of interest]
> >
> > git provides many ways of modifying the branch (and the rebase topic is
> very generic) so I think its best to rely on proper git docs/tutorials
> > [and its not really specific to petsc workflow]
>
> You can do it in one line, without changing the base:
>
>   git rebase -i $(git merge-base main HEAD)
>
>
Oh, this has a side effect that can not be undone with some sort of simple
--abort.

Not for me and  Blaise.


> An alternative is
>
>   git rebase -i main
>
> which gives you interactive rebase to replay on top of current 'main'.
> This does two things at once and changing the base for your branch is not
> always desirable.
>

Humm, sounds interesting. I'll try it. Looks simple.

Re: [petsc-dev] Commit squashing in MR

[Mark Adams]

On Tue, Mar 2, 2021 at 11:09 PM Jacob Faibussowitsch 
wrote:

>   I do not get this. I thought that rebasing with main put all the main
> changes in before your commits. I have never seen any interspersed, so I do
> not understand this.
>
>
> Mark, do you rebase your branch over main first or do you merge main intro
> your branch to update it?
>

I never merge anymore. just rebase over main.


> If you rebase, you can pick an N such that you only get your commits from
> the branch.
>

Yea, but what is N? If you know that then use it. I just guess if I lose
track (I bounce between my Mac and two GPU machines so I can do a lot of
wip commits) and as long as you guess too big, you will see where your
commits start and ignore the rest.


>
> Best regards,
>
> Jacob Faibussowitsch
> (Jacob Fai - booss - oh - vitch)
> Cell: (312) 694-3391
>
> On Mar 2, 2021, at 22:02, Barry Smith  wrote:
>
>
>
> On Mar 2, 2021, at 9:24 PM, Mark Adams  wrote:
>
> Ah, 'git graph' I will try that next time.
>
> I use 'git rebase -i HEAD~N', but you need an N.
>
> After you 'git rebase origin/main' you get other commits interspersed in
> with yours,
>
>
>I do not get this. I thought that rebasing with main put all the main
> changes in before your commits. I have never seen any interspersed, so I do
> not understand this.
>
>   Barry
>
> so I try to rebase -i before rebasing over main. Then rebase over main and
> you have a clean and updated branch.
>
> Pick N to be large enough to cover the commits that you want to clean up.
> Don't touch the ones that are not yours from main, the last time you
> rebased over main.
>
> On Tue, Mar 2, 2021 at 10:02 PM Junchao Zhang 
> wrote:
>
>> I am a naive git user, so I use interactive git rebase.  Suppose I am on
>> the branch I want to modify,
>>
>> 1) Use git graph to locate an upstream commit to be used as the base
>> $ git graph
>> * 0d5433e9 (HEAD -> jczhang/sf-change-api) SF: rename SFCreateEmbeddedSF
>> to SFCreateEmbeddedRootSF
>> * e7314fbb SF: add an MPI_Op argument to SFBcast
>> * 83df288d Replace MPIU_REPLACE with MPI_REPLACE
>> *   b434c516 Merge branch
>> 'barry/2021-02-02/petscsf-communication-specific' into 'main'
>> |\
>> | * 62152ded (barry/2021-02-02/petscsf-communication-specific)
>> PetscSFView() never called viewer for the specific type (bug), hence many
>> output files were incorrect.
>> * |   a4f5d9b4 Merge branch 'jose/upgrade-magma' into 'main'
>>
>> 2) Suppose we choose b434c516 as the base. All commits we want to squash
>> are after it.  Do interactive git rebase. It shows a screen for you to
>> edit.  Read the help, which is helpful for new users
>>   $ git rebase -i b434c516
>> pick 83df288d Replace MPIU_REPLACE with MPI_REPLACE
>> pick e7314fbb SF: add an MPI_Op argument to SFBcast
>> pick 0d5433e9 SF: rename SFCreateEmbeddedSF to SFCreateEmbeddedRootSF
>>
>> # Rebase b434c516..0d5433e9 onto b434c516 (3 commands)
>> #
>> # Commands:
>> # p, pick  = use commit
>> # r, reword  = use commit, but edit the commit message
>> # e, edit  = use commit, but stop for amending
>> # s, squash  = use commit, but meld into previous commit
>> # f, fixup  = like "squash", but discard this commit's log message
>> # x, exec  = run command (the rest of the line) using shell
>> # b, break = stop here (continue rebase later with 'git rebase
>> --continue')
>> # d, drop  = remove commit
>> # l, label  = label current HEAD with a name
>> # t, reset  = reset HEAD to a label
>> # m, merge [-C  | -c ]  [# ]
>> # .   create a merge commit using the original merge commit's
>> # .   message (or the oneline, if no original merge commit was
>> # .   specified). Use -c  to reword the commit message.
>> #
>> # These lines can be re-ordered; they are executed from top to bottom.
>> #
>> # If you remove a line here THAT COMMIT WILL BE LOST.
>> #
>> # However, if you remove everything, the rebase will be aborted.
>> #
>> # Note that empty commits are commented out
>>
>> 3) Suppose we want to squash the last two commits to 83df288d, replace
>> their pick with s (or f, see the help for difference), save and exit the
>> screen
>> pick 83df288d Replace MPIU_REPLACE with MPI_REPLACE
>> s e7314fbb SF: add an MPI_Op argument to SFBcast
>> s 0d5433e9 SF: rename SFCreateEmbeddedSF to SFCreateEmbeddedRootSF
>>
>> A new screen shows up
>>
>> # This is a combination of 3 commits.
>> # This is the 1st commit message:
>>
>> Replace MPIU_REPLACE with MPI_REPLACE
>>
>> Since we believe all MPI implementations support MPI_REPLACE
>>
>> # This is the commit message #2:
>>
>> SF: add an MPI_Op argument to SFBcast
>>
>> # This is the commit message #3:
>>
>> SF: rename SFCreateEmbeddedSF to SFCreateEmbeddedRootSF
>>
>> # Please enter the commit message for your changes. Lines starting
>> # with '#' will be ignored, and an empty message aborts the commit.
>>
>> 4) Edit the commit message as you want, save and exit, done!
>>
>> --Junchao Zhang
>>
>>
>> On Tue, Mar 2, 2021 at 6:19 PM Blaise A Bourdin  

Re: [petsc-dev] Case TS005062693 - XLF: ICE in xlfentry compiling a module with 358 subroutines

[Satish Balay via petsc-dev]

The only change I can get working (i.e avoid compile errors) is to split 
petscvecmod.F90 into 2 source files - but I don't know if this will help with 
xlf..

My [partial] change is in branch balay/reorg-f90-for-xlf

Satish

On Wed, 3 Mar 2021, Satish Balay via petsc-dev wrote:

> On Wed, 3 Mar 2021, Satish Balay via petsc-dev wrote:
> 
> > Sure - once any change works locally [for gcc and xlf]
> > 
> > When I try - I get a bunch of errors.. [yet to digest them.]
> 
> > > >>> Can you please give the following source code workaround a try?
> > > >>> Since there is already "use petscvecdefdummy" at the module scope, 
> > > >>> one workaround might be to remove the unnecessary "use 
> > > >>> petscvecdefdummy" in vecnotequal and vecequals 
> > > >>> and all similar procedures.
> > > >>> 
> > > >>> For example, the test case has:
> > > >>>module petscvecdef
> > > >>>use petscvecdefdummy
> > > >>> ...
> > > >>>function vecnotequal(A,B)
> > > >>>  use petscvecdefdummy
> > > >>>  logical vecnotequal
> > > >>>  type(tVec), intent(in) :: A,B
> > > >>>  vecnotequal = (A%v .ne. B%v)
> > > >>>end function
> 
> 
> Ok - try this suggestion:
> 
> diff --git a/src/vec/f90-mod/petscvecmod.F90 b/src/vec/f90-mod/petscvecmod.F90
> index 0c447156b9..81968c7ca1 100644
> --- a/src/vec/f90-mod/petscvecmod.F90
> +++ b/src/vec/f90-mod/petscvecmod.F90
> @@ -77,7 +77,6 @@
>  use petscvecdefdummy
>  interface operator(.ne.)
>function vecnotequal(A,B)
> -use petscvecdefdummy
>  logical vecnotequal
>  type(tVec), intent(in) :: A,B
>end function
> 
> 
> >
>   FC arch-linux-c-debug/obj/vec/f90-mod/petscvecmod.o
> /home/balay/petsc/src/vec/f90-mod/petscvecmod.F90:81:22:
> 
>81 | type(tVec), intent(in) :: A,B
>   |  1
> Error: Derived type ‘tvec’ at (1) is being used before it is defined
> /home/balay/petsc/include/../src/vec/f90-mod/ftn-auto-interfaces/petscis.h90:2:10:
> 
> 2 |   use petscvecdef
>   |  1
> Fatal Error: Cannot open module file ‘petscvecdef.mod’ for reading at (1): No 
> such file or directory
> <<
> 
> Satish

Re: [petsc-dev] Commit squashing in MR

[Mark Adams]

On Tue, Mar 2, 2021 at 11:02 PM Barry Smith  wrote:

>
>
> On Mar 2, 2021, at 9:24 PM, Mark Adams  wrote:
>
> Ah, 'git graph' I will try that next time.
>
> I use 'git rebase -i HEAD~N', but you need an N.
>
> After you 'git rebase origin/main' you get other commits interspersed in
> with yours,
>
>
>I do not get this. I thought that rebasing with main put all the main
> changes in before your commits. I have never seen any interspersed, so I do
> not understand this.
>

Humm, I'm not sure. The important thing is that you will hit main commits
and you don't want to touch those or the first of your commits.


>
>   Barry
>
> so I try to rebase -i before rebasing over main. Then rebase over main and
> you have a clean and updated branch.
>
> Pick N to be large enough to cover the commits that you want to clean up.
> Don't touch the ones that are not yours from main, the last time you
> rebased over main.
>
> On Tue, Mar 2, 2021 at 10:02 PM Junchao Zhang 
> wrote:
>
>> I am a naive git user, so I use interactive git rebase.  Suppose I am on
>> the branch I want to modify,
>>
>> 1) Use git graph to locate an upstream commit to be used as the base
>> $ git graph
>> * 0d5433e9 (HEAD -> jczhang/sf-change-api) SF: rename SFCreateEmbeddedSF
>> to SFCreateEmbeddedRootSF
>> * e7314fbb SF: add an MPI_Op argument to SFBcast
>> * 83df288d Replace MPIU_REPLACE with MPI_REPLACE
>> *   b434c516 Merge branch
>> 'barry/2021-02-02/petscsf-communication-specific' into 'main'
>> |\
>> | * 62152ded (barry/2021-02-02/petscsf-communication-specific)
>> PetscSFView() never called viewer for the specific type (bug), hence many
>> output files were incorrect.
>> * |   a4f5d9b4 Merge branch 'jose/upgrade-magma' into 'main'
>>
>> 2) Suppose we choose b434c516 as the base. All commits we want to squash
>> are after it.  Do interactive git rebase. It shows a screen for you to
>> edit.  Read the help, which is helpful for new users
>>   $ git rebase -i b434c516
>> pick 83df288d Replace MPIU_REPLACE with MPI_REPLACE
>> pick e7314fbb SF: add an MPI_Op argument to SFBcast
>> pick 0d5433e9 SF: rename SFCreateEmbeddedSF to SFCreateEmbeddedRootSF
>>
>> # Rebase b434c516..0d5433e9 onto b434c516 (3 commands)
>> #
>> # Commands:
>> # p, pick  = use commit
>> # r, reword  = use commit, but edit the commit message
>> # e, edit  = use commit, but stop for amending
>> # s, squash  = use commit, but meld into previous commit
>> # f, fixup  = like "squash", but discard this commit's log message
>> # x, exec  = run command (the rest of the line) using shell
>> # b, break = stop here (continue rebase later with 'git rebase
>> --continue')
>> # d, drop  = remove commit
>> # l, label  = label current HEAD with a name
>> # t, reset  = reset HEAD to a label
>> # m, merge [-C  | -c ]  [# ]
>> # .   create a merge commit using the original merge commit's
>> # .   message (or the oneline, if no original merge commit was
>> # .   specified). Use -c  to reword the commit message.
>> #
>> # These lines can be re-ordered; they are executed from top to bottom.
>> #
>> # If you remove a line here THAT COMMIT WILL BE LOST.
>> #
>> # However, if you remove everything, the rebase will be aborted.
>> #
>> # Note that empty commits are commented out
>>
>> 3) Suppose we want to squash the last two commits to 83df288d, replace
>> their pick with s (or f, see the help for difference), save and exit the
>> screen
>> pick 83df288d Replace MPIU_REPLACE with MPI_REPLACE
>> s e7314fbb SF: add an MPI_Op argument to SFBcast
>> s 0d5433e9 SF: rename SFCreateEmbeddedSF to SFCreateEmbeddedRootSF
>>
>> A new screen shows up
>>
>> # This is a combination of 3 commits.
>> # This is the 1st commit message:
>>
>> Replace MPIU_REPLACE with MPI_REPLACE
>>
>> Since we believe all MPI implementations support MPI_REPLACE
>>
>> # This is the commit message #2:
>>
>> SF: add an MPI_Op argument to SFBcast
>>
>> # This is the commit message #3:
>>
>> SF: rename SFCreateEmbeddedSF to SFCreateEmbeddedRootSF
>>
>> # Please enter the commit message for your changes. Lines starting
>> # with '#' will be ignored, and an empty message aborts the commit.
>>
>> 4) Edit the commit message as you want, save and exit, done!
>>
>> --Junchao Zhang
>>
>>
>> On Tue, Mar 2, 2021 at 6:19 PM Blaise A Bourdin  wrote:
>>
>>> Hi,
>>>
>>> This is not technically a petsc question.
>>> It would be great to have a short section in the PETSc integration
>>> workflow document explaining how to squash commits in a MR for git-impaired
>>> developers like me.
>>>
>>> Anybody wants to pitch in, or explain me how to do this?
>>>
>>> Regards,
>>> Blaise
>>>
>>> --
>>> A.K. & Shirley Barton Professor of  Mathematics
>>> Adjunct Professor of Mechanical Engineering
>>> Adjunct of the Center for Computation & Technology
>>> Louisiana State University, Lockett Hall Room 344, Baton Rouge, LA
>>> 70803, USA
>>> Tel. +1 (225) 578 1612, Fax  +1 (225) 578 4276 Web
>>> http://www.math.lsu.edu/~bourdin
>>>
>>>
>

Re: [petsc-dev] Case TS005062693 - XLF: ICE in xlfentry compiling a module with 358 subroutines

[Satish Balay via petsc-dev]

On Wed, 3 Mar 2021, Satish Balay via petsc-dev wrote:

> Sure - once any change works locally [for gcc and xlf]
> 
> When I try - I get a bunch of errors.. [yet to digest them.]

> > >>> Can you please give the following source code workaround a try?
> > >>> Since there is already "use petscvecdefdummy" at the module scope, one 
> > >>> workaround might be to remove the unnecessary "use petscvecdefdummy" in 
> > >>> vecnotequal and vecequals 
> > >>> and all similar procedures.
> > >>> 
> > >>> For example, the test case has:
> > >>>module petscvecdef
> > >>>use petscvecdefdummy
> > >>> ...
> > >>>function vecnotequal(A,B)
> > >>>  use petscvecdefdummy
> > >>>  logical vecnotequal
> > >>>  type(tVec), intent(in) :: A,B
> > >>>  vecnotequal = (A%v .ne. B%v)
> > >>>end function


Ok - try this suggestion:

diff --git a/src/vec/f90-mod/petscvecmod.F90 b/src/vec/f90-mod/petscvecmod.F90
index 0c447156b9..81968c7ca1 100644
--- a/src/vec/f90-mod/petscvecmod.F90
+++ b/src/vec/f90-mod/petscvecmod.F90
@@ -77,7 +77,6 @@
 use petscvecdefdummy
 interface operator(.ne.)
   function vecnotequal(A,B)
-use petscvecdefdummy
 logical vecnotequal
 type(tVec), intent(in) :: A,B
   end function


>
  FC arch-linux-c-debug/obj/vec/f90-mod/petscvecmod.o
/home/balay/petsc/src/vec/f90-mod/petscvecmod.F90:81:22:

   81 | type(tVec), intent(in) :: A,B
  |  1
Error: Derived type ‘tvec’ at (1) is being used before it is defined
/home/balay/petsc/include/../src/vec/f90-mod/ftn-auto-interfaces/petscis.h90:2:10:

2 |   use petscvecdef
  |  1
Fatal Error: Cannot open module file ‘petscvecdef.mod’ for reading at (1): No 
such file or directory
<<

Satish

Re: [petsc-dev] Case TS005062693 - XLF: ICE in xlfentry compiling a module with 358 subroutines

[Satish Balay via petsc-dev]

Sure - once any change works locally [for gcc and xlf]

When I try - I get a bunch of errors.. [yet to digest them.]

Satish

On Wed, 3 Mar 2021, Jacob Faibussowitsch wrote:

> > I'm not sure what would happen if these 'use' statements are removed [whats 
> > required and what can be removed?]
> > 
> > The relevant code that adds this is in 
> > lib/petsc/bin/maint/generatefortranstubs.py
> > 
> >  fd.write('  use petsc'+mansec+'def\n')
> 
> I suppose we can run it through CI, see if it breaks? 
> 
> Best regards,
> 
> Jacob Faibussowitsch
> (Jacob Fai - booss - oh - vitch)
> Cell: (312) 694-3391
> 
> > On Mar 3, 2021, at 12:49, Satish Balay  wrote:
> > 
> > On Wed, 3 Mar 2021, Jacob Faibussowitsch wrote:
> > 
> >> Hello All,
> >> 
> >> I discovered a compiler bug in the IBM xl fortran compiler a few weeks ago 
> >> that would crash the compiler when compiling petsc fortran interfaces. The 
> >> TL;DR of it is that the xl compiler creates a function dictionary for 
> >> every function imported in fortran modules, and since petsc fortran 
> >> interfaces seem to import entire packages writ-large this exceeds the 
> >> number of dictionary entries (2**21):
> >> 
> >>> The reason for the Internal Compiler Error is because we can't grow an 
> >>> interal dictionary anymore (ie we hit a 2**21 limit).
> >>> The file contains many module procedures and interfaces that use the same 
> >>> helper module. As a result, we are importing the dictionary entries for 
> >>> that module repeatedly reaching 
> >>> the limit.
> >>> 
> >>> Can you please give the following source code workaround a try?
> >>> Since there is already "use petscvecdefdummy" at the module scope, one 
> >>> workaround might be to remove the unnecessary "use petscvecdefdummy" in 
> >>> vecnotequal and vecequals 
> >>> and all similar procedures.
> >>> 
> >>> For example, the test case has:
> >>>module petscvecdef
> >>>use petscvecdefdummy
> >>> ...
> >>>function vecnotequal(A,B)
> >>>  use petscvecdefdummy
> >>>  logical vecnotequal
> >>>  type(tVec), intent(in) :: A,B
> >>>  vecnotequal = (A%v .ne. B%v)
> >>>end function
> >>>function vecequals(A,B)
> >>>  use petscvecdefdummy
> >>>  logical vecequals
> >>>  type(tVec), intent(in) :: A,B
> >>>  vecequals = (A%v .eq. B%v)
> >>>end function
> >>> ...
> >>> end module
> >>> Another workaround would be to put the procedure definitions from this 
> >>> large module into several submodules.  Each submodule would be able to 
> >>> accommodate a dictionary with 2**21 entries.
> >>> 
> >>> 
> >>> Please let us know if one of the above workarounds resolve the issue.
> >> 
> >> 
> >> The proposed fix from IBM would be to pull “use moduleXXX” out of 
> >> subroutines or to have our auto-fortran interfaces detect which symbols to 
> >> include from the respective modules and only include those in the 
> >> subroutines. I’m not familiar at all with how the interfaces are generated 
> >> so I don’t even know if this is possible.
> > 
> > I'm not sure what would happen if these 'use' statements are removed [whats 
> > required and what can be removed?]
> > 
> > The relevant code that adds this is in 
> > lib/petsc/bin/maint/generatefortranstubs.py
> > 
> >  fd.write('  use petsc'+mansec+'def\n')
> > 
> > Satish
> > 
> >>> IBM provided the following additional explanation and example. Can the 
> >>> process used to generate these routines and functions determine the 
> >>> specific symbols required and then use the only keyword or import 
> >>> statement to include them?
> >>> 
> >>> When factoring out use statements out of module procedures, you can just 
> >>> delete them.  But you can't completely remove them from interface blocks. 
> >>>  Instead, you can limit them either by using use , only:  
> >>> or import  . if the hundreds of use statements in the program are 
> >>> factored out / limited in this way, that should reduce the dictionary 
> >>> size sufficiently for the program to compile.
> >>> 
> >>> For example
> >>>  Interface
> >>>Subroutine VecRestoreArrayReadF90(v,array,ierr)
> >>>  use petscvecdef
> >>>  real(kind=selected_real_kind(10)), pointer :: array(:)
> >>>  integer(kind=selected_int_kind(5)) ierr
> >>>  type(tVec) v
> >>>End Subroutine
> >>>  End Interface
> >>> 
> >>> imports all symbols from petscvecdef into the dictionary even though we 
> >>> only need tVec .  So we can either:
> >>> 
> >>>  Interface
> >>>Subroutine VecRestoreArrayReadF90(v,array,ierr)
> >>>  use petscvecdef, only: tVec
> >>>  implicit none
> >>>  real(kind=selected_real_kind(10)), pointer :: array(:)
> >>>  integer(kind=selected_int_kind(5)) ierr
> >>>  type(tVec) v
> >>>End Subroutine
> >>>  End Interface
> >>> 
> >>> or if use petscvecdef is used in 

Re: [petsc-dev] Case TS005062693 - XLF: ICE in xlfentry compiling a module with 358 subroutines

[Jacob Faibussowitsch]

> I'm not sure what would happen if these 'use' statements are removed [whats 
> required and what can be removed?]
> 
> The relevant code that adds this is in 
> lib/petsc/bin/maint/generatefortranstubs.py
> 
>  fd.write('  use petsc'+mansec+'def\n')

I suppose we can run it through CI, see if it breaks? 

Best regards,

Jacob Faibussowitsch
(Jacob Fai - booss - oh - vitch)
Cell: (312) 694-3391

> On Mar 3, 2021, at 12:49, Satish Balay  wrote:
> 
> On Wed, 3 Mar 2021, Jacob Faibussowitsch wrote:
> 
>> Hello All,
>> 
>> I discovered a compiler bug in the IBM xl fortran compiler a few weeks ago 
>> that would crash the compiler when compiling petsc fortran interfaces. The 
>> TL;DR of it is that the xl compiler creates a function dictionary for every 
>> function imported in fortran modules, and since petsc fortran interfaces 
>> seem to import entire packages writ-large this exceeds the number of 
>> dictionary entries (2**21):
>> 
>>> The reason for the Internal Compiler Error is because we can't grow an 
>>> interal dictionary anymore (ie we hit a 2**21 limit).
>>> The file contains many module procedures and interfaces that use the same 
>>> helper module. As a result, we are importing the dictionary entries for 
>>> that module repeatedly reaching 
>>> the limit.
>>> 
>>> Can you please give the following source code workaround a try?
>>> Since there is already "use petscvecdefdummy" at the module scope, one 
>>> workaround might be to remove the unnecessary "use petscvecdefdummy" in 
>>> vecnotequal and vecequals 
>>> and all similar procedures.
>>> 
>>> For example, the test case has:
>>>module petscvecdef
>>>use petscvecdefdummy
>>> ...
>>>function vecnotequal(A,B)
>>>  use petscvecdefdummy
>>>  logical vecnotequal
>>>  type(tVec), intent(in) :: A,B
>>>  vecnotequal = (A%v .ne. B%v)
>>>end function
>>>function vecequals(A,B)
>>>  use petscvecdefdummy
>>>  logical vecequals
>>>  type(tVec), intent(in) :: A,B
>>>  vecequals = (A%v .eq. B%v)
>>>end function
>>> ...
>>> end module
>>> Another workaround would be to put the procedure definitions from this 
>>> large module into several submodules.  Each submodule would be able to 
>>> accommodate a dictionary with 2**21 entries.
>>> 
>>> 
>>> Please let us know if one of the above workarounds resolve the issue.
>> 
>> 
>> The proposed fix from IBM would be to pull “use moduleXXX” out of 
>> subroutines or to have our auto-fortran interfaces detect which symbols to 
>> include from the respective modules and only include those in the 
>> subroutines. I’m not familiar at all with how the interfaces are generated 
>> so I don’t even know if this is possible.
> 
> I'm not sure what would happen if these 'use' statements are removed [whats 
> required and what can be removed?]
> 
> The relevant code that adds this is in 
> lib/petsc/bin/maint/generatefortranstubs.py
> 
>  fd.write('  use petsc'+mansec+'def\n')
> 
> Satish
> 
>>> IBM provided the following additional explanation and example. Can the 
>>> process used to generate these routines and functions determine the 
>>> specific symbols required and then use the only keyword or import statement 
>>> to include them?
>>> 
>>> When factoring out use statements out of module procedures, you can just 
>>> delete them.  But you can't completely remove them from interface blocks.  
>>> Instead, you can limit them either by using use , only:  or 
>>> import  . if the hundreds of use statements in the program are 
>>> factored out / limited in this way, that should reduce the dictionary size 
>>> sufficiently for the program to compile.
>>> 
>>> For example
>>>  Interface
>>>Subroutine VecRestoreArrayReadF90(v,array,ierr)
>>>  use petscvecdef
>>>  real(kind=selected_real_kind(10)), pointer :: array(:)
>>>  integer(kind=selected_int_kind(5)) ierr
>>>  type(tVec) v
>>>End Subroutine
>>>  End Interface
>>> 
>>> imports all symbols from petscvecdef into the dictionary even though we 
>>> only need tVec .  So we can either:
>>> 
>>>  Interface
>>>Subroutine VecRestoreArrayReadF90(v,array,ierr)
>>>  use petscvecdef, only: tVec
>>>  implicit none
>>>  real(kind=selected_real_kind(10)), pointer :: array(:)
>>>  integer(kind=selected_int_kind(5)) ierr
>>>  type(tVec) v
>>>End Subroutine
>>>  End Interface
>>> 
>>> or if use petscvecdef is used in the outer scope, we can:
>>>  Interface
>>>Subroutine VecRestoreArrayReadF90(v,array,ierr)
>>>  import tVec
>>>  implicit none
>>>  real(kind=selected_real_kind(10)), pointer :: array(:)
>>>  integer(kind=selected_int_kind(5)) ierr
>>>  type(tVec) v
>>>End Subroutine
>>>  End Interface
>>> (The two methods (use, only vs import) are equivalent 

Re: [petsc-dev] Fwd: Case TS005062693 - XLF: ICE in xlfentry compiling a module with 358 subroutines

[Satish Balay via petsc-dev]

On Wed, 3 Mar 2021, Jacob Faibussowitsch wrote:

> Hello All,
> 
> I discovered a compiler bug in the IBM xl fortran compiler a few weeks ago 
> that would crash the compiler when compiling petsc fortran interfaces. The 
> TL;DR of it is that the xl compiler creates a function dictionary for every 
> function imported in fortran modules, and since petsc fortran interfaces seem 
> to import entire packages writ-large this exceeds the number of dictionary 
> entries (2**21):
> 
> > The reason for the Internal Compiler Error is because we can't grow an 
> > interal dictionary anymore (ie we hit a 2**21 limit).
> > The file contains many module procedures and interfaces that use the same 
> > helper module. As a result, we are importing the dictionary entries for 
> > that module repeatedly reaching 
> > the limit.
> >  
> > Can you please give the following source code workaround a try?
> > Since there is already "use petscvecdefdummy" at the module scope, one 
> > workaround might be to remove the unnecessary "use petscvecdefdummy" in 
> > vecnotequal and vecequals 
> > and all similar procedures.
> >  
> > For example, the test case has:
> > module petscvecdef
> > use petscvecdefdummy
> > ...
> > function vecnotequal(A,B)
> >   use petscvecdefdummy
> >   logical vecnotequal
> >   type(tVec), intent(in) :: A,B
> >   vecnotequal = (A%v .ne. B%v)
> > end function
> > function vecequals(A,B)
> >   use petscvecdefdummy
> >   logical vecequals
> >   type(tVec), intent(in) :: A,B
> >   vecequals = (A%v .eq. B%v)
> > end function
> > ...
> > end module
> > Another workaround would be to put the procedure definitions from this 
> > large module into several submodules.  Each submodule would be able to 
> > accommodate a dictionary with 2**21 entries.
> >  
> >  
> > Please let us know if one of the above workarounds resolve the issue.
> 
> 
> The proposed fix from IBM would be to pull “use moduleXXX” out of subroutines 
> or to have our auto-fortran interfaces detect which symbols to include from 
> the respective modules and only include those in the subroutines. I’m not 
> familiar at all with how the interfaces are generated so I don’t even know if 
> this is possible.

I'm not sure what would happen if these 'use' statements are removed [whats 
required and what can be removed?]

The relevant code that adds this is in 
lib/petsc/bin/maint/generatefortranstubs.py

  fd.write('  use petsc'+mansec+'def\n')

Satish

> > IBM provided the following additional explanation and example. Can the 
> > process used to generate these routines and functions determine the 
> > specific symbols required and then use the only keyword or import statement 
> > to include them?
> > 
> >  When factoring out use statements out of module procedures, you can just 
> > delete them.  But you can't completely remove them from interface blocks.  
> > Instead, you can limit them either by using use , only:  or 
> > import  .  if the hundreds of use statements in the program are 
> > factored out / limited in this way, that should reduce the dictionary size 
> > sufficiently for the program to compile.
> >  
> > For example
> >   Interface
> > Subroutine VecRestoreArrayReadF90(v,array,ierr)
> >   use petscvecdef
> >   real(kind=selected_real_kind(10)), pointer :: array(:)
> >   integer(kind=selected_int_kind(5)) ierr
> >   type(tVec) v
> > End Subroutine
> >   End Interface
> >  
> > imports all symbols from petscvecdef into the dictionary even though we 
> > only need tVec .  So we can either:
> >  
> >   Interface
> > Subroutine VecRestoreArrayReadF90(v,array,ierr)
> >   use petscvecdef, only: tVec
> >   implicit none
> >   real(kind=selected_real_kind(10)), pointer :: array(:)
> >   integer(kind=selected_int_kind(5)) ierr
> >   type(tVec) v
> > End Subroutine
> >   End Interface
> >  
> > or if use petscvecdef is used in the outer scope, we can:
> >   Interface
> > Subroutine VecRestoreArrayReadF90(v,array,ierr)
> >   import tVec
> >   implicit none
> >   real(kind=selected_real_kind(10)), pointer :: array(:)
> >   integer(kind=selected_int_kind(5)) ierr
> >   type(tVec) v
> > End Subroutine
> >   End Interface
> > (The two methods (use, only vs import) are equivalent in terms of impact to 
> > the dictionary.)
> > 
> 
> Is this compiler ~feature~ something that we intend to work around? Thoughts?
> 
> Best regards,
> 
> Jacob Faibussowitsch
> (Jacob Fai - booss - oh - vitch)
> Cell: (312) 694-3391
> 
> > Begin forwarded message:
> > 
> > From: "Roy Musselman" 
> > Subject: Re: Case TS005062693 - XLF: ICE in xlfentry compiling a module 
> > with 358 subroutines
> > Date: March 3, 2021 at 08:23:17 CST
> > To: Jacob 

Re: [petsc-dev] Fwd: Case TS005062693 - XLF: ICE in xlfentry compiling a module with 358 subroutines

[Satish Balay via petsc-dev]

On Wed, 3 Mar 2021, Jacob Faibussowitsch wrote:

> Hello All,
> 
> I discovered a compiler bug in the IBM xl fortran compiler a few weeks ago 
> that would crash the compiler when compiling petsc fortran interfaces. The 
> TL;DR of it is that the xl compiler creates a function dictionary for every 
> function imported in fortran modules, and since petsc fortran interfaces seem 
> to import entire packages writ-large this exceeds the number of dictionary 
> entries (2**21):
> 
> > The reason for the Internal Compiler Error is because we can't grow an 
> > interal dictionary anymore (ie we hit a 2**21 limit).
> > The file contains many module procedures and interfaces that use the same 
> > helper module. As a result, we are importing the dictionary entries for 
> > that module repeatedly reaching 
> > the limit.
> >  
> > Can you please give the following source code workaround a try?
> > Since there is already "use petscvecdefdummy" at the module scope, one 
> > workaround might be to remove the unnecessary "use petscvecdefdummy" in 
> > vecnotequal and vecequals 
> > and all similar procedures.

This sounds reasonable  - but the change might be tedious [to make without 
breaking some required dependency]. Perhaps it will also help gfortran RAM 
requirements..

Satish

> >  
> > For example, the test case has:
> > module petscvecdef
> > use petscvecdefdummy
> > ...
> > function vecnotequal(A,B)
> >   use petscvecdefdummy
> >   logical vecnotequal
> >   type(tVec), intent(in) :: A,B
> >   vecnotequal = (A%v .ne. B%v)
> > end function
> > function vecequals(A,B)
> >   use petscvecdefdummy
> >   logical vecequals
> >   type(tVec), intent(in) :: A,B
> >   vecequals = (A%v .eq. B%v)
> > end function
> > ...
> > end module
> > Another workaround would be to put the procedure definitions from this 
> > large module into several submodules.  Each submodule would be able to 
> > accommodate a dictionary with 2**21 entries.
> >  
> >  
> > Please let us know if one of the above workarounds resolve the issue.
> 
> 
> The proposed fix from IBM would be to pull “use moduleXXX” out of subroutines 
> or to have our auto-fortran interfaces detect which symbols to include from 
> the respective modules and only include those in the subroutines. I’m not 
> familiar at all with how the interfaces are generated so I don’t even know if 
> this is possible.
> > IBM provided the following additional explanation and example. Can the 
> > process used to generate these routines and functions determine the 
> > specific symbols required and then use the only keyword or import statement 
> > to include them?
> > 
> >  When factoring out use statements out of module procedures, you can just 
> > delete them.  But you can't completely remove them from interface blocks.  
> > Instead, you can limit them either by using use , only:  or 
> > import  .  if the hundreds of use statements in the program are 
> > factored out / limited in this way, that should reduce the dictionary size 
> > sufficiently for the program to compile.
> >  
> > For example
> >   Interface
> > Subroutine VecRestoreArrayReadF90(v,array,ierr)
> >   use petscvecdef
> >   real(kind=selected_real_kind(10)), pointer :: array(:)
> >   integer(kind=selected_int_kind(5)) ierr
> >   type(tVec) v
> > End Subroutine
> >   End Interface
> >  
> > imports all symbols from petscvecdef into the dictionary even though we 
> > only need tVec .  So we can either:
> >  
> >   Interface
> > Subroutine VecRestoreArrayReadF90(v,array,ierr)
> >   use petscvecdef, only: tVec
> >   implicit none
> >   real(kind=selected_real_kind(10)), pointer :: array(:)
> >   integer(kind=selected_int_kind(5)) ierr
> >   type(tVec) v
> > End Subroutine
> >   End Interface
> >  
> > or if use petscvecdef is used in the outer scope, we can:
> >   Interface
> > Subroutine VecRestoreArrayReadF90(v,array,ierr)
> >   import tVec
> >   implicit none
> >   real(kind=selected_real_kind(10)), pointer :: array(:)
> >   integer(kind=selected_int_kind(5)) ierr
> >   type(tVec) v
> > End Subroutine
> >   End Interface
> > (The two methods (use, only vs import) are equivalent in terms of impact to 
> > the dictionary.)
> > 
> 
> Is this compiler ~feature~ something that we intend to work around? Thoughts?
> 
> Best regards,
> 
> Jacob Faibussowitsch
> (Jacob Fai - booss - oh - vitch)
> Cell: (312) 694-3391
> 
> > Begin forwarded message:
> > 
> > From: "Roy Musselman" 
> > Subject: Re: Case TS005062693 - XLF: ICE in xlfentry compiling a module 
> > with 358 subroutines
> > Date: March 3, 2021 at 08:23:17 CST
> > To: Jacob Faibussowitsch 
> > Cc: "Gyllenhaal, John C." 
> > 
> > Hi Jacob, 
> > I tried the first 

Re: [petsc-dev] Case TS005062693 - XLF: ICE in xlfentry compiling a module with 358 subroutines

[Jed Brown]

I wonder if gfortran has a similar "bug" but larger capacity that would explain 
why it's so much more expensive to compile the Fortran interface files than to 
compile all of PETSc.

Barry Smith  writes:

>PETSc stacks the Fortran modules in the same way it stacks the C include 
> files.   So the TAO module includes all the Fortran modules below it etc.  It 
> would be nearly impossible to disentangle the bits and pieces without 
> introducing a more painful user experience. For example use PCTypes, use 
> PCFunctions, use KSPTypes,  impossible to use and impossible to maintain.
>
>This is a completely artificial bug of IBM's own making in their compiler 
> that we should not have to work around.
>
>Barry
>
>
>> On Mar 3, 2021, at 12:10 PM, Jacob Faibussowitsch  
>> wrote:
>> 
>> Hello All,
>> 
>> I discovered a compiler bug in the IBM xl fortran compiler a few weeks ago 
>> that would crash the compiler when compiling petsc fortran interfaces. The 
>> TL;DR of it is that the xl compiler creates a function dictionary for every 
>> function imported in fortran modules, and since petsc fortran interfaces 
>> seem to import entire packages writ-large this exceeds the number of 
>> dictionary entries (2**21):
>> 
>>> The reason for the Internal Compiler Error is because we can't grow an 
>>> interal dictionary anymore (ie we hit a 2**21 limit).
>>> The file contains many module procedures and interfaces that use the same 
>>> helper module. As a result, we are importing the dictionary entries for 
>>> that module repeatedly reaching 
>>> the limit.
>>>  
>>> Can you please give the following source code workaround a try?
>>> Since there is already "use petscvecdefdummy" at the module scope, one 
>>> workaround might be to remove the unnecessary "use petscvecdefdummy" in 
>>> vecnotequal and vecequals 
>>> and all similar procedures.
>>>  
>>> For example, the test case has:
>>> module petscvecdef
>>> use petscvecdefdummy
>>> ...
>>> function vecnotequal(A,B)
>>>   use petscvecdefdummy
>>>   logical vecnotequal
>>>   type(tVec), intent(in) :: A,B
>>>   vecnotequal = (A%v .ne. B%v)
>>> end function
>>> function vecequals(A,B)
>>>   use petscvecdefdummy
>>>   logical vecequals
>>>   type(tVec), intent(in) :: A,B
>>>   vecequals = (A%v .eq. B%v)
>>> end function
>>> ...
>>> end module
>>> Another workaround would be to put the procedure definitions from this 
>>> large module into several submodules.  Each submodule would be able to 
>>> accommodate a dictionary with 2**21 entries.
>>>  
>>>  
>>> Please let us know if one of the above workarounds resolve the issue.
>> 
>> 
>> The proposed fix from IBM would be to pull “use moduleXXX” out of 
>> subroutines or to have our auto-fortran interfaces detect which symbols to 
>> include from the respective modules and only include those in the 
>> subroutines. I’m not familiar at all with how the interfaces are generated 
>> so I don’t even know if this is possible.
>>> IBM provided the following additional explanation and example. Can the 
>>> process used to generate these routines and functions determine the 
>>> specific symbols required and then use the only keyword or import statement 
>>> to include them?
>>> 
>>>  When factoring out use statements out of module procedures, you can just 
>>> delete them.  But you can't completely remove them from interface blocks.  
>>> Instead, you can limit them either by using use , only:  or 
>>> import  .  if the hundreds of use statements in the program are 
>>> factored out / limited in this way, that should reduce the dictionary size 
>>> sufficiently for the program to compile.
>>>  
>>> For example
>>>   Interface
>>> Subroutine VecRestoreArrayReadF90(v,array,ierr)
>>>   use petscvecdef
>>>   real(kind=selected_real_kind(10)), pointer :: array(:)
>>>   integer(kind=selected_int_kind(5)) ierr
>>>   type(tVec) v
>>> End Subroutine
>>>   End Interface
>>>  
>>> imports all symbols from petscvecdef into the dictionary even though we 
>>> only need tVec .  So we can either:
>>>  
>>>   Interface
>>> Subroutine VecRestoreArrayReadF90(v,array,ierr)
>>>   use petscvecdef, only: tVec
>>>   implicit none
>>>   real(kind=selected_real_kind(10)), pointer :: array(:)
>>>   integer(kind=selected_int_kind(5)) ierr
>>>   type(tVec) v
>>> End Subroutine
>>>   End Interface
>>>  
>>> or if use petscvecdef is used in the outer scope, we can:
>>>   Interface
>>> Subroutine VecRestoreArrayReadF90(v,array,ierr)
>>>   import tVec
>>>   implicit none
>>>   real(kind=selected_real_kind(10)), pointer :: array(:)
>>>   integer(kind=selected_int_kind(5)) ierr
>>>   type(tVec) v
>>> End Subroutine
>>>   End Interface
>>> (The 

Re: [petsc-dev] Case TS005062693 - XLF: ICE in xlfentry compiling a module with 358 subroutines

[Barry Smith]

   PETSc stacks the Fortran modules in the same way it stacks the C include 
files.   So the TAO module includes all the Fortran modules below it etc.  It 
would be nearly impossible to disentangle the bits and pieces without 
introducing a more painful user experience. For example use PCTypes, use 
PCFunctions, use KSPTypes,  impossible to use and impossible to maintain.

   This is a completely artificial bug of IBM's own making in their compiler 
that we should not have to work around.

   Barry


> On Mar 3, 2021, at 12:10 PM, Jacob Faibussowitsch  wrote:
> 
> Hello All,
> 
> I discovered a compiler bug in the IBM xl fortran compiler a few weeks ago 
> that would crash the compiler when compiling petsc fortran interfaces. The 
> TL;DR of it is that the xl compiler creates a function dictionary for every 
> function imported in fortran modules, and since petsc fortran interfaces seem 
> to import entire packages writ-large this exceeds the number of dictionary 
> entries (2**21):
> 
>> The reason for the Internal Compiler Error is because we can't grow an 
>> interal dictionary anymore (ie we hit a 2**21 limit).
>> The file contains many module procedures and interfaces that use the same 
>> helper module. As a result, we are importing the dictionary entries for that 
>> module repeatedly reaching 
>> the limit.
>>  
>> Can you please give the following source code workaround a try?
>> Since there is already "use petscvecdefdummy" at the module scope, one 
>> workaround might be to remove the unnecessary "use petscvecdefdummy" in 
>> vecnotequal and vecequals 
>> and all similar procedures.
>>  
>> For example, the test case has:
>> module petscvecdef
>> use petscvecdefdummy
>> ...
>> function vecnotequal(A,B)
>>   use petscvecdefdummy
>>   logical vecnotequal
>>   type(tVec), intent(in) :: A,B
>>   vecnotequal = (A%v .ne. B%v)
>> end function
>> function vecequals(A,B)
>>   use petscvecdefdummy
>>   logical vecequals
>>   type(tVec), intent(in) :: A,B
>>   vecequals = (A%v .eq. B%v)
>> end function
>> ...
>> end module
>> Another workaround would be to put the procedure definitions from this large 
>> module into several submodules.  Each submodule would be able to accommodate 
>> a dictionary with 2**21 entries.
>>  
>>  
>> Please let us know if one of the above workarounds resolve the issue.
> 
> 
> The proposed fix from IBM would be to pull “use moduleXXX” out of subroutines 
> or to have our auto-fortran interfaces detect which symbols to include from 
> the respective modules and only include those in the subroutines. I’m not 
> familiar at all with how the interfaces are generated so I don’t even know if 
> this is possible.
>> IBM provided the following additional explanation and example. Can the 
>> process used to generate these routines and functions determine the specific 
>> symbols required and then use the only keyword or import statement to 
>> include them?
>> 
>>  When factoring out use statements out of module procedures, you can just 
>> delete them.  But you can't completely remove them from interface blocks.  
>> Instead, you can limit them either by using use , only:  or 
>> import  .  if the hundreds of use statements in the program are 
>> factored out / limited in this way, that should reduce the dictionary size 
>> sufficiently for the program to compile.
>>  
>> For example
>>   Interface
>> Subroutine VecRestoreArrayReadF90(v,array,ierr)
>>   use petscvecdef
>>   real(kind=selected_real_kind(10)), pointer :: array(:)
>>   integer(kind=selected_int_kind(5)) ierr
>>   type(tVec) v
>> End Subroutine
>>   End Interface
>>  
>> imports all symbols from petscvecdef into the dictionary even though we only 
>> need tVec .  So we can either:
>>  
>>   Interface
>> Subroutine VecRestoreArrayReadF90(v,array,ierr)
>>   use petscvecdef, only: tVec
>>   implicit none
>>   real(kind=selected_real_kind(10)), pointer :: array(:)
>>   integer(kind=selected_int_kind(5)) ierr
>>   type(tVec) v
>> End Subroutine
>>   End Interface
>>  
>> or if use petscvecdef is used in the outer scope, we can:
>>   Interface
>> Subroutine VecRestoreArrayReadF90(v,array,ierr)
>>   import tVec
>>   implicit none
>>   real(kind=selected_real_kind(10)), pointer :: array(:)
>>   integer(kind=selected_int_kind(5)) ierr
>>   type(tVec) v
>> End Subroutine
>>   End Interface
>> (The two methods (use, only vs import) are equivalent in terms of impact to 
>> the dictionary.)
>> 
> 
> Is this compiler ~feature~ something that we intend to work around? Thoughts?
> 
> Best regards,
> 
> Jacob Faibussowitsch
> (Jacob Fai - booss - oh - vitch)
> Cell: (312) 694-3391
> 
>> Begin forwarded message:
>> 
>> From: "Roy 

[petsc-dev] Fwd: Case TS005062693 - XLF: ICE in xlfentry compiling a module with 358 subroutines

[Jacob Faibussowitsch]

Hello All,

I discovered a compiler bug in the IBM xl fortran compiler a few weeks ago that 
would crash the compiler when compiling petsc fortran interfaces. The TL;DR of 
it is that the xl compiler creates a function dictionary for every function 
imported in fortran modules, and since petsc fortran interfaces seem to import 
entire packages writ-large this exceeds the number of dictionary entries 
(2**21):

> The reason for the Internal Compiler Error is because we can't grow an 
> interal dictionary anymore (ie we hit a 2**21 limit).
> The file contains many module procedures and interfaces that use the same 
> helper module. As a result, we are importing the dictionary entries for that 
> module repeatedly reaching 
> the limit.
>  
> Can you please give the following source code workaround a try?
> Since there is already "use petscvecdefdummy" at the module scope, one 
> workaround might be to remove the unnecessary "use petscvecdefdummy" in 
> vecnotequal and vecequals 
> and all similar procedures.
>  
> For example, the test case has:
> module petscvecdef
> use petscvecdefdummy
> ...
> function vecnotequal(A,B)
>   use petscvecdefdummy
>   logical vecnotequal
>   type(tVec), intent(in) :: A,B
>   vecnotequal = (A%v .ne. B%v)
> end function
> function vecequals(A,B)
>   use petscvecdefdummy
>   logical vecequals
>   type(tVec), intent(in) :: A,B
>   vecequals = (A%v .eq. B%v)
> end function
> ...
> end module
> Another workaround would be to put the procedure definitions from this large 
> module into several submodules.  Each submodule would be able to accommodate 
> a dictionary with 2**21 entries.
>  
>  
> Please let us know if one of the above workarounds resolve the issue.


The proposed fix from IBM would be to pull “use moduleXXX” out of subroutines 
or to have our auto-fortran interfaces detect which symbols to include from the 
respective modules and only include those in the subroutines. I’m not familiar 
at all with how the interfaces are generated so I don’t even know if this is 
possible.
> IBM provided the following additional explanation and example. Can the 
> process used to generate these routines and functions determine the specific 
> symbols required and then use the only keyword or import statement to include 
> them?
> 
>  When factoring out use statements out of module procedures, you can just 
> delete them.  But you can't completely remove them from interface blocks.  
> Instead, you can limit them either by using use , only:  or 
> import  .  if the hundreds of use statements in the program are 
> factored out / limited in this way, that should reduce the dictionary size 
> sufficiently for the program to compile.
>  
> For example
>   Interface
> Subroutine VecRestoreArrayReadF90(v,array,ierr)
>   use petscvecdef
>   real(kind=selected_real_kind(10)), pointer :: array(:)
>   integer(kind=selected_int_kind(5)) ierr
>   type(tVec) v
> End Subroutine
>   End Interface
>  
> imports all symbols from petscvecdef into the dictionary even though we only 
> need tVec .  So we can either:
>  
>   Interface
> Subroutine VecRestoreArrayReadF90(v,array,ierr)
>   use petscvecdef, only: tVec
>   implicit none
>   real(kind=selected_real_kind(10)), pointer :: array(:)
>   integer(kind=selected_int_kind(5)) ierr
>   type(tVec) v
> End Subroutine
>   End Interface
>  
> or if use petscvecdef is used in the outer scope, we can:
>   Interface
> Subroutine VecRestoreArrayReadF90(v,array,ierr)
>   import tVec
>   implicit none
>   real(kind=selected_real_kind(10)), pointer :: array(:)
>   integer(kind=selected_int_kind(5)) ierr
>   type(tVec) v
> End Subroutine
>   End Interface
> (The two methods (use, only vs import) are equivalent in terms of impact to 
> the dictionary.)
> 

Is this compiler ~feature~ something that we intend to work around? Thoughts?

Best regards,

Jacob Faibussowitsch
(Jacob Fai - booss - oh - vitch)
Cell: (312) 694-3391

> Begin forwarded message:
> 
> From: "Roy Musselman" 
> Subject: Re: Case TS005062693 - XLF: ICE in xlfentry compiling a module with 
> 358 subroutines
> Date: March 3, 2021 at 08:23:17 CST
> To: Jacob Faibussowitsch 
> Cc: "Gyllenhaal, John C." 
> 
> Hi Jacob, 
> I tried the first suggestion and commented out the use statements called 
> within the functions. However, I hit the following error complaining about 
> specific symbol dependencies provided by the library.
> 
> .../src/vec/f90-mod/petscvecmod.F90", line 107.37: 1514-084 (S) Identifier a 
> is being declared with type name tvec which has not been defined in a derived 
> type definition. 
> 
> IBM provided the following additional explanation and example. Can the 
> process used to 

Re: [petsc-dev] Petsc "make test" have more failures for --with-openmp=1

[Barry Smith]

>> 2) I can reproduce the src/mat/tests/ex242.c error (which explicitly uses 
>> ScaLAPACK, none of the above PC uses it explicitly, except PCBDDC/PCHPDDM 
>> when using MUMPS on “big” problems where root nodes are factorized using 
>> ScaLAPACK, see -mat_mumps_icntl_13)
>> 3) I’m seeing that both on your machine and mine, PETSc BuildSystem insist 
>> on linking libmkl_blacs_intelmpi_lp64.so even though we supply explicitly 
>> libmkl_blacs_openmpi_lp64.so
>> This for example yields a wrong Makefile.inc for MUMPS:
>> $ cat arch-linux2-c-opt-ompi/externalpackages/MUMPS_5.3.5/Makefile.inc|grep 
>> blacs
>> SCALAP  = […] -lmkl_blacs_openmpi_lp64
>> LIBBLAS = […] -lmkl_blacs_intelmpi_lp64 -lgomp -ldl -lpthread -lm […]
>> 
>> Despite what Barry says, I think PETSc is partially to blame as well (why 
>> use libmkl_blacs_intelmpi_lp64.so even though BuildSystem is capable of 
>> detecting we are using OpenMPI).
>> I’ll try to fix this to see if it solves 2).
> Okay, that's a very nice finding!!!  Hope it will be "fixable" easily!
> 
> 
The knowledge is there but the information may not be trivially available to 
make the right decisions. Parts of the BLAS/LAPACK checks use the "check 
everything" approach. For example

# Look for Multi-Threaded MKL for MKL_C/Pardiso
  useCPardiso=0
  usePardiso=0
  if self.argDB['with-mkl_cpardiso'] or 'with-mkl_cpardiso-dir' in 
self.argDB or 'with-mkl_cpardiso-lib' in self.argDB:
useCPardiso=1

mkl_blacs_64=[['mkl_blacs_intelmpi'+ILP64+''],['mkl_blacs_mpich'+ILP64+''],['mkl_blacs_sgimpt'+ILP64+''],['mkl_blacs_openmpi'+ILP64+'']]

mkl_blacs_32=[['mkl_blacs_intelmpi'],['mkl_blacs_mpich'],['mkl_blacs_sgimpt'],['mkl_blacs_openmpi']]
  elif self.argDB['with-mkl_pardiso'] or 'with-mkl_pardiso-dir' in 
self.argDB or 'with-mkl_pardiso-lib' in self.argDB:
usePardiso=1
mkl_blacs_64=[[]]
mkl_blacs_32=[[]]
  if useCPardiso or usePardiso:
self.logPrintBox('BLASLAPACK: Looking for Multithreaded MKL for 
C/Pardiso')
for libdir in 
[os.path.join('lib','64'),os.path.join('lib','ia64'),os.path.join('lib','em64t'),os.path.join('lib','intel64'),'lib','64','ia64','em64t','intel64',
   
os.path.join('lib','32'),os.path.join('lib','ia32'),'32','ia32','']:
  if not os.path.exists(os.path.join(dir,libdir)):
self.logPrint('MKL Path not found.. skipping: 
'+os.path.join(dir,libdir))
  else:
self.log.write('Files and directories in that 
directory:\n'+str(os.listdir(os.path.join(dir,libdir)))+'\n')
#  iomp5 is provided by the Intel compilers on MacOS. Run source 
/opt/intel/bin/compilervars.sh intel64 to have it added to LIBRARY_PATH
#  then locate libimp5.dylib in the LIBRARY_PATH and copy it to 
os.path.join(dir,libdir)
for i in mkl_blacs_64:
  yield ('User specified MKL-C/Pardiso Intel-Linux64', None, 
[os.path.join(dir,libdir,'libmkl_intel'+ILP64+'.a'),'mkl_core','mkl_intel_thread']+i+['iomp5','dl','pthread'],known,'yes')
  yield ('User specified MKL-C/Pardiso GNU-Linux64', None, 
[os.path.join(dir,libdir,'libmkl_intel'+ILP64+'.a'),'mkl_core','mkl_gnu_thread']+i+['gomp','dl','pthread'],known,'yes')
  yield ('User specified MKL-Pardiso Intel-Windows64', None, 
[os.path.join(dir,libdir,'mkl_core.lib'),'mkl_intel'+ILP64+'.lib','mkl_intel_thread.lib']+i+['libiomp5md.lib'],known,'yes')
for i in mkl_blacs_32:
  yield ('User specified MKL-C/Pardiso Intel-Linux32', None, 
[os.path.join(dir,libdir,'libmkl_intel.a'),'mkl_core','mkl_intel_thread']+i+['iomp5','dl','pthread'],'32','yes')
  yield ('User specified MKL-C/Pardiso GNU-Linux32', None, 
[os.path.join(dir,libdir,'libmkl_intel.a'),'mkl_core','mkl_gnu_thread']+i+['gomp','dl','pthread'],'32','yes')
  yield ('User specified MKL-Pardiso Intel-Windows32', None, 
[os.path.join(dir,libdir,'mkl_core.lib'),'mkl_intel_c.lib','mkl_intel_thread.lib']+i+['libiomp5md.lib'],'32','yes')
return

The assumption is that the link will fail unless the correct libraries are in 
the list. But apparently this is not the case; it returns the first case that 
links but that case does not run which is why it appears to be producing 
"silly" results.

If you set the right MPI and threading library, at these locations instead of 
trying all of them it might resolve the problems. 

if self.openmp.found:
  ITHREAD='intel_thread'
  ITHREADGNU='gnu_thread'
  ompthread = 'yes'
else:
  ITHREAD='sequential'
  ITHREADGNU='sequential'
  ompthread = 'no'


mkl_blacs_64=[['mkl_blacs_intelmpi'+ILP64+''],['mkl_blacs_mpich'+ILP64+''],['mkl_blacs_sgimpt'+ILP64+''],['mkl_blacs_openmpi'+ILP64+'']]

mkl_blacs_32=[['mkl_blacs_intelmpi'],['mkl_blacs_mpich'],['mkl_blacs_sgimpt'],['mkl_blacs_openmpi']]




> 


> On Mar 3, 2021, at 8:22 AM, Eric Chamberland 
>  wrote:
> 
> Hi Pierre,
> 
> 

Re: [petsc-dev] Commit squashing in MR

[Zhang, Hong via petsc-dev]

Patrick,
I need update petsc manual on DMNetwork, but do not know how to proceed. I 
tried your suggested steps:
1) go to the docs page you want to edit on 
docs.petsc.org
2) select the version you want (usually "main") in the black ReadTheDocs box in 
the lower right
3) click "edit" in "on GitLab" and make your MR (name the branch with "docs-" 
to maybe get it to auto-build on ReadTheDocs, label with docs and docs-only)

I do not understand 3). Can you give a tutorial demo in next petsc meeting?
Hong

From: petsc-dev  on behalf of Patrick Sanan 

Sent: Wednesday, March 3, 2021 12:23 AM
To: Jed Brown 
Cc: Satish Balay via petsc-dev 
Subject: Re: [petsc-dev] Commit squashing in MR

The whole section on git in the dev manual needs some attention. (It was moved 
there in the consolidation of docs we had scattered in various places, but 
hasn't been expertly updated yet). Ideal, I think, would be to find some good, 
external instructions and link to them, under the idea that we should only 
maintain things in our own docs that aren't adequately documented somewhere 
else. This might not be possible (since we had to create these instructions in 
the first place).

There is a section on squashing but it's currently a bit buried, and the advice 
in this thread is probably more useful/current
https://docs.petsc.org/en/main/developers/integration/#squashing-excessive-commits

If anyone wants to go in there and quickly update those docs, remember that you 
can do so all from web interfaces! This workflow still has some wrinkles, but 
for small changes I still think it's appealing:

- go to the docs page you want to edit on docs.petsc.org
- select the version you want (usually "main") in the black ReadTheDocs box in 
the lower right
- click "edit" in "on GitLab" and make your MR (name the branch with "docs-" to 
maybe get it to auto-build on ReadTheDocs, label with docs and docs-only)
- if you get feedback on your MR and need to update, or notice a typo, I 
*think* this will work:
   - click on the last commit of your new branch
   - find the offending file
   - click on "edit at @deadbeef123"
- change the branch *back* to your branch in the pulldown
- click "edit"
- back in your MR, edit to "squash commits"

You can get a partial preview with the usual "preview" button, though not 
everything is interpreted correctly (but for things like links, it works fine).

If you want a full preview, you can

1. Build the Sphinx docs locally from your branch, either with
- "make sphinx-docs-all LOC=$PETSC_DIR"  (you may need to add PYTHON=python3, 
since this relies on Python 3.3+ for venv)
- install the required Python packages yourself (e.g. pip install -r 
src/docs/sphinx_docs/requirements.txt), go to src/docs/sphinx_docs, run "make 
html", and look in _build/html

2. Build the Sphinx docs for your branch as a version on ReadTheDocs. There is 
currently an automation rule there that if your branch name has "docs-" in it, 
it should build (though I must admit I'm still not completely sure I understand 
exactly when RTD updates its information from GitLab). Or, if you have access, 
you can activate a new version yourself.



Am 03.03.2021 um 05:32 schrieb Jed Brown 
mailto:j...@jedbrown.org>>:

Satish Balay via petsc-dev 
mailto:petsc-dev@mcs.anl.gov>> writes:

On Wed, 3 Mar 2021, Blaise A Bourdin wrote:

Hi,

This is not technically a petsc question.
It would be great to have a short section in the PETSc integration workflow 
document explaining how to squash commits in a MR for git-impaired developers 
like me.

Anybody wants to pitch in, or explain me how to do this?

To squash commits - I use the 'squash' action in 'git rebase -i HASH' and 
figure out the HASH to use from 'gitk main..branch'

[as git rebase requires the commit prior to the first commit of interest]

git provides many ways of modifying the branch (and the rebase topic is very 
generic) so I think its best to rely on proper git docs/tutorials
[and its not really specific to petsc workflow]

You can do it in one line, without changing the base:

 git rebase -i $(git merge-base main HEAD)


An alternative is

 git rebase -i main

which gives you interactive rebase to replay on top of current 'main'. This 
does two things at once and changing the base for your branch is not always 
desirable.

Re: [petsc-dev] Petsc "make test" have more failures for --with-openmp=1

[Eric Chamberland]

Hi Pierre,

On 2021-03-03 2:42 a.m., Pierre Jolivet wrote:


If it ends that there is a problem combining MKL + openMP that relies 
on linking configuration for example, should it be a good thing to 
have this (--with-openmp=1) tested into the pipelines (with external 
packages of course)?



As Barry said, there is not much (if any) OpenMP in PETSc.
There is however some workers with the MKL (+ Intel compilers) turned 
on, but I don’t think we test MKL + GNU compilers (which I feel like 
is a very niche combination, hence not really worth testing, IMHO).


Ouch, this is my almost my personal working configuration and for most 
of our users too... and it worked well until I activated the OpenMP thing...


We had good reasons to work with g++ or clang++ instead of intel compilers:

- It is mandatory to pay to work with an intel compiler (didn't looked 
at OneAPI licensing yet, but it may have changed?)


- No support of Intel compilers with iceccd (slow recompilation)

- MKL was freely distributed, so it can be used with any compiler

That doesn't mean we don't want to use intel compiler, but maybe we just 
want to to a specific delivery with it but continue to develop with g++ 
or clang++ (my personal choice).


But I understand it is less straightforward to combine gcc and MKL than 
using native Intel tool-chain



Does the guys who maintain all these libs are reading petsc-dev? ;)

I don’t think they are, but don’t worry, we do forward the appropriate 
messages to them :)

:)


About yesterday’s failures…
1) I cannot reproduce any of the PCHYPRE/PCBDDC/PCHPDDM errors (sorry 
I didn’t bother putting the SuperLU_DIST tarball on my cluster)


Hmmm, maybe my environment variables may play a role into this?

for comparisons considerations, we explicitly set:

export MKL_CBWR=COMPATIBLE
export MKL_NUM_THREADS=1

but it would be surprising it helps reproduce a problem: they usually 
stabilize results...


2) I can reproduce the src/mat/tests/ex242.c error (which explicitly 
uses ScaLAPACK, none of the above PC uses it explicitly, except 
PCBDDC/PCHPDDM when using MUMPS on “big” problems where root nodes are 
factorized using ScaLAPACK, see -mat_mumps_icntl_13)
3) I’m seeing that both on your machine and mine, PETSc BuildSystem 
insist on linking libmkl_blacs_intelmpi_lp64.so even though we supply 
explicitly libmkl_blacs_openmpi_lp64.so

This for example yields a wrong Makefile.inc for MUMPS:
$ cat 
arch-linux2-c-opt-ompi/externalpackages/MUMPS_5.3.5/Makefile.inc|grep 
blacs

SCALAP  = […] -lmkl_blacs_openmpi_lp64
LIBBLAS = […] -lmkl_blacs_intelmpi_lp64 -lgomp -ldl -lpthread -lm […]

Despite what Barry says, I think PETSc is partially to blame as well 
(why use libmkl_blacs_intelmpi_lp64.so even though BuildSystem is 
capable of detecting we are using OpenMPI).

I’ll try to fix this to see if it solves 2).


Okay, that's a very nice finding!!!  Hope it will be "fixable" easily!

Merci,

Eric



Thanks,
Pierre

http://joliv.et/irene-rome-configure.log 


$ /usr/bin/gmake -f gmakefile test test-fail=1
Using MAKEFLAGS: test-fail=1
        TEST 
arch-linux2-c-opt-ompi/tests/counts/snes_tutorials-ex12_quad_hpddm_reuse_baij.counts

 ok snes_tutorials-ex12_quad_hpddm_reuse_baij
 ok diff-snes_tutorials-ex12_quad_hpddm_reuse_baij
        TEST 
arch-linux2-c-opt-ompi/tests/counts/ksp_ksp_tutorials-ex50_tut_2.counts
 ok ksp_ksp_tutorials-ex50_tut_2 # SKIP PETSC_HAVE_SUPERLU_DIST 
requirement not met
        TEST 
arch-linux2-c-opt-ompi/tests/counts/snes_tutorials-ex56_hypre.counts

 ok snes_tutorials-ex56_hypre
 ok diff-snes_tutorials-ex56_hypre
        TEST 
arch-linux2-c-opt-ompi/tests/counts/snes_tutorials-ex17_3d_q3_trig_elas.counts

 ok snes_tutorials-ex17_3d_q3_trig_elas
 ok diff-snes_tutorials-ex17_3d_q3_trig_elas
        TEST 
arch-linux2-c-opt-ompi/tests/counts/snes_tutorials-ex12_quad_hpddm_reuse_threshold_baij.counts

 ok snes_tutorials-ex12_quad_hpddm_reuse_threshold_baij
 ok diff-snes_tutorials-ex12_quad_hpddm_reuse_threshold_baij
        TEST 
arch-linux2-c-opt-ompi/tests/counts/snes_tutorials-ex12_tri_parmetis_hpddm_baij.counts

 ok snes_tutorials-ex12_tri_parmetis_hpddm_baij
 ok diff-snes_tutorials-ex12_tri_parmetis_hpddm_baij
        TEST 
arch-linux2-c-opt-ompi/tests/counts/snes_tutorials-ex19_tut_3.counts

 ok snes_tutorials-ex19_tut_3
 ok diff-snes_tutorials-ex19_tut_3
        TEST arch-linux2-c-opt-ompi/tests/counts/mat_tests-ex242_3.counts
not ok mat_tests-ex242_3 # Error code: 137
#[1]PETSC ERROR: 

#[1]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, 
probably memory access out of range
#[1]PETSC ERROR: Try option -start_in_debugger or 
-on_error_attach_debugger
#[1]PETSC ERROR: or see 
https://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind 

#[1]PETSC ERROR: or try http://valgrind.org 

Re: [petsc-dev] Petsc "make test" have more failures for --with-openmp=1

[Pierre Jolivet]

Here is a fix for PETSc disregarding what kind of MPI implementation is being 
used with cluster PARDISO: https://gitlab.com/petsc/petsc/-/merge_requests/3678 

This doesn’t fix src/mat/tests/ex242, whose backtrace looks suspicious to me…
(gdb) #0  0x2ac152c9c6ed in mkl_trans_avx2_mkl_domatcopy2_t ()
   from 
/ccc/products/mkl-19.0.5.281/intel--19.0.5.281__openmpi--4.0.1/default/19.0.5.281/mkl/lib/intel64/libmkl_avx2.so
#1  0x2ba345400dc7 in mkl_trans_avx2_mkl_domatcopy ()
   from 
/ccc/products/mkl-19.0.5.281/intel--19.0.5.281__openmpi--4.0.1/default/19.0.5.281/mkl/lib/intel64/libmkl_avx2.so
#2  0x2ba31564fac2 in dmmdatdl_ ()
   from 
/ccc/products/mkl-19.0.5.281/intel--19.0.5.281__openmpi--4.0.1/default/19.0.5.281/mkl/lib/intel64/libmkl_scalapack_lp64.so
#3  0x2ba3156d70de in PB_Cptran_DC ()
   from 
/ccc/products/mkl-19.0.5.281/intel--19.0.5.281__openmpi--4.0.1/default/19.0.5.281/mkl/lib/intel64/libmkl_scalapack_lp64.so
#4  0x2ba315740062 in pdtran_ ()
   from 
/ccc/products/mkl-19.0.5.281/intel--19.0.5.281__openmpi--4.0.1/default/19.0.5.281/mkl/lib/intel64/libmkl_scalapack_lp64.so
#5  0x2ba31370211b in MatTranspose_ScaLAPACK ()
   from 
/ccc/work/cont003/rndm/rndm/petsc/arch-linux2-c-opt-ompi/lib/libpetsc.so.3.014

I’m now realizing that this is also segfaulting with BLACS IntelMPI, so it’s 
probably unrelated, and simply some Intel mess.
(gdb) #0  0x2ab818c4f3de in mkl_trans_def_mkl_domatcopy2_t ()
   from 
/ccc/products/mkl-20.0.4/system/default/20.0.4/mkl/lib/intel64/libmkl_def.so
#1  0x2ab818c51247 in mkl_trans_def_mkl_domatcopy ()
   from 
/ccc/products/mkl-20.0.4/system/default/20.0.4/mkl/lib/intel64/libmkl_def.so
#2  0x2ab76e0ba062 in dmmdatdl_ ()
   from 
/ccc/products/mkl-20.0.4/system/default/20.0.4/mkl/lib/intel64/libmkl_scalapack_lp64.so
#3  0x2ab76e14157e in PB_Cptran_DC ()
   from 
/ccc/products/mkl-20.0.4/system/default/20.0.4/mkl/lib/intel64/libmkl_scalapack_lp64.so
#4  0x2ab76e1aa502 in pdtran_ ()
   from 
/ccc/products/mkl-20.0.4/system/default/20.0.4/mkl/lib/intel64/libmkl_scalapack_lp64.so
#5  0x2b7f87c833a1 in MatTranspose_ScaLAPACK ()
   from 
/ccc/work/cont003/rndm/rndm/petsc/arch-linux2-c-opt-impi/lib/libpetsc.so.3.014

(Notice the switch from arch-linux2-c-opt-ompi to arch-linux2-c-opt-impi)

Thanks,
Pierre

> On 3 Mar 2021, at 8:42 AM, Pierre Jolivet  wrote:
> 
> 
>> If it ends that there is a problem combining MKL + openMP that relies on 
>> linking configuration for example, should it be a good thing to have this 
>> (--with-openmp=1) tested into the pipelines (with external packages of 
>> course)?
>> 
> As Barry said, there is not much (if any) OpenMP in PETSc.
> There is however some workers with the MKL (+ Intel compilers) turned on, but 
> I don’t think we test MKL + GNU compilers (which I feel like is a very niche 
> combination, hence not really worth testing, IMHO).
> 
>> Does the guys who maintain all these libs are reading petsc-dev? ;)
>> 
> I don’t think they are, but don’t worry, we do forward the appropriate 
> messages to them :)
> 
> About yesterday’s failures…
> 1) I cannot reproduce any of the PCHYPRE/PCBDDC/PCHPDDM errors (sorry I 
> didn’t bother putting the SuperLU_DIST tarball on my cluster)
> 2) I can reproduce the src/mat/tests/ex242.c error (which explicitly uses 
> ScaLAPACK, none of the above PC uses it explicitly, except PCBDDC/PCHPDDM 
> when using MUMPS on “big” problems where root nodes are factorized using 
> ScaLAPACK, see -mat_mumps_icntl_13)
> 3) I’m seeing that both on your machine and mine, PETSc BuildSystem insist on 
> linking libmkl_blacs_intelmpi_lp64.so even though we supply explicitly 
> libmkl_blacs_openmpi_lp64.so
> This for example yields a wrong Makefile.inc for MUMPS:
> $ cat arch-linux2-c-opt-ompi/externalpackages/MUMPS_5.3.5/Makefile.inc|grep 
> blacs
> SCALAP  = […] -lmkl_blacs_openmpi_lp64
> LIBBLAS = […] -lmkl_blacs_intelmpi_lp64 -lgomp -ldl -lpthread -lm […]
> 
> Despite what Barry says, I think PETSc is partially to blame as well (why use 
> libmkl_blacs_intelmpi_lp64.so even though BuildSystem is capable of detecting 
> we are using OpenMPI).
> I’ll try to fix this to see if it solves 2).
> 
> Thanks,
> Pierre
> 
> http://joliv.et/irene-rome-configure.log 
> 
> $ /usr/bin/gmake -f gmakefile test test-fail=1
> Using MAKEFLAGS: test-fail=1
> TEST 
> arch-linux2-c-opt-ompi/tests/counts/snes_tutorials-ex12_quad_hpddm_reuse_baij.counts
>  ok snes_tutorials-ex12_quad_hpddm_reuse_baij
>  ok diff-snes_tutorials-ex12_quad_hpddm_reuse_baij
> TEST 
> arch-linux2-c-opt-ompi/tests/counts/ksp_ksp_tutorials-ex50_tut_2.counts
>  ok ksp_ksp_tutorials-ex50_tut_2 # SKIP PETSC_HAVE_SUPERLU_DIST requirement 
> not met
> TEST 
> arch-linux2-c-opt-ompi/tests/counts/snes_tutorials-ex56_hypre.counts
>  ok snes_tutorials-ex56_hypre
>  ok diff-snes_tutorials-ex56_hypre
> 

Re: [petsc-dev] problem registering a new solver

[Mark Adams]

On Tue, Mar 2, 2021 at 5:22 PM Matthew Knepley  wrote:

> Maybe there is a bug in the Register() code.
>

Humm, but I just add one more registration 

 ierr = MatSolverTypeRegister(MATSOLVERPETSC, MATSEQAIJ,
 MAT_FACTOR_LU,MatGetFactor_seqaij_petsc);CHKERRQ(ierr);
  ierr = MatSolverTypeRegister(MATSOLVERPETSC, MATSEQAIJ,
 MAT_FACTOR_CHOLESKY,MatGetFactor_seqaij_petsc);CHKERRQ(ierr);
  ierr = MatSolverTypeRegister(MATSOLVERPETSC, MATSEQAIJ,
 MAT_FACTOR_ILU,MatGetFactor_seqaij_petsc);CHKERRQ(ierr);
  ierr = MatSolverTypeRegister(MATSOLVERPETSC, MATSEQAIJ,
 MAT_FACTOR_ICC,MatGetFactor_seqaij_petsc);CHKERRQ(ierr);

*  ierr = MatSolverTypeRegister(MATSOLVERPETSC, MATSEQAIJ,
 MAT_FACTOR_LUBAND,MatGetFactor_seqaij_petsc);CHKERRQ(ierr);*

luband is a little different in that it does not touch PC, there is no
-pc_type luband (now), while the others are pc_types. But that should not
matter.

Thanks,


>Matt
>
> On Tue, Mar 2, 2021 at 4:43 PM Mark Adams  wrote:
>
>> I see the problem, but not the solution. I put a print statement
>> in MatSolverTypeDestroy and see:
>>
>> MatSolverTypeDestroy seqaij inext->next=0x7fb4a9114c70
>> inext=0x7fb4a9113c70
>>
>> before it fails. This seqaij node seems to be pointing to itself. If I
>> remove my registration call and it works and I see this from my print
>> statement:
>>
>> MatSolverTypeDestroy seqaij inext->next=0x7fe811827270
>> inext=0x7fe811826270
>> MatSolverTypeDestroy seqaijperm inext->next=0x7fe811909c70
>> inext=0x7fe811827270
>> MatSolverTypeDestroy constantdiagonal inext->next=0x7fe81190ac70
>> inext=0x7fe811909c70
>>  
>>
>>
>> On Tue, Mar 2, 2021 at 3:42 PM Mark Adams  wrote:
>>
>>> I am trying to add a band solver to PETSc (later to be moved to Cuda and
>>> Kokkos) and I have started by adding some types and a copy of the current
>>> LU as placeholders. I register with:
>>>
>>>  ierr = MatSolverTypeRegister(MATSOLVERPETSC, MATSEQAIJ,
>>>  MAT_FACTOR_LUBAND,MatGetFactor_seqaij_petsc);CHKERRQ(ierr);
>>>
>>> And that is about all that I do that can have any effect at this point.
>>> I add a switch in  MatGetFactor_seqaij_petsc on (ftype ==
>>> MAT_FACTOR_LUBAND) to set the symbolic factorization
>>> method (*B)->ops->lufactorsymbolic  = MatLUBandFactorSymbolic_SeqAIJ; But
>>> this is not called (I verified this) because I don't know how to
>>> get MatGetFactor_seqaij_petsc to receive ftype == MAT_FACTOR_LUBAND. (I
>>> need help on this to)
>>>
>>> Anyway, I would hope that these changes would not do anything but I get
>>> an error (appended).
>>>
>>> It is failing in MatSolverTypeDestroy on this second PetscFree:
>>>
>>> while (inext) {
>>>   ierr = PetscFree(inext->mtype);CHKERRQ(ierr);
>>>   iprev = inext;
>>>   inext = inext->next;
>>>   ierr = PetscFree(iprev);CHKERRQ(ierr);
>>> }
>>>
>>> I tried to clone LU here, but I clearly missed something.
>>>
>>> Any ideas?
>>>
>>> And I just made  an MR for this if you want to look at the code.
>>>
>>> Thanks,
>>> Mark
>>> ...
>>> Number of SNES iterations = 2
>>> [0]PETSC ERROR: PetscTrFreeDefault() called from MatSolverTypeDestroy()
>>> line 4513 in /Users/markadams/Codes/petsc/src/mat/interface/matrix.c
>>> [0]PETSC ERROR: Block [id=0(48)] at address 0x7fb7fb8f7620 is corrupted
>>> (probably write past end of array)
>>> [0]PETSC ERROR: Block allocated in MatSolverTypeRegister() line 4382 in
>>> /Users/markadams/Codes/petsc/src/mat/interface/matrix.c
>>> [0]PETSC ERROR: - Error Message
>>> --
>>> [0]PETSC ERROR: Memory corruption:
>>> https://www.mcs.anl.gov/petsc/documentation/installation.html#valgrind
>>> [0]PETSC ERROR: Corrupted memory
>>> [0]PETSC ERROR: See https://www.mcs.anl.gov/petsc/documentation/faq.html
>>> for trouble shooting.
>>> [0]PETSC ERROR: Petsc Development GIT revision: v3.14.4-739-g575f931ef8
>>>  GIT Date: 2021-03-02 13:38:55 -0500
>>> [0]PETSC ERROR: ./ex19 on a arch-macosx-gnu-g named MarksMac-302.local
>>> by markadams Tue Mar  2 15:28:41 2021
>>> [0]PETSC ERROR: Configure options
>>> --with-mpi-dir=/usr/local/Cellar/mpich/3.3.2_1 COPTFLAGS="-g -O0"
>>> CXXOPTFLAGS="-g -O0" --download-metis=1 --download-parmetis=1
>>> --download-kokkos=1 --download-kokkos-kernels=1 --download-p4est=1
>>> --with-zlib=1 --download-superlu_dist --download-superlu --with-make-np=4
>>> --download-hdf5=1 -with-cuda=0 --with-x=0 --with-debugging=1
>>> PETSC_ARCH=arch-macosx-gnu-g --with-64-bit-indices=0 --with-openmp=0
>>> --with-ctable=0
>>> [0]PETSC ERROR: #1 PetscTrFreeDefault() line 310 in
>>> /Users/markadams/Codes/petsc/src/sys/memory/mtr.c
>>> [0]PETSC ERROR: #2 MatSolverTypeDestroy() line 4513 in
>>> /Users/markadams/Codes/petsc/src/mat/interface/matrix.c
>>> [0]PETSC ERROR: #3 MatFinalizePackage() line 57 in
>>> /Users/markadams/Codes/petsc/src/mat/interface/dlregismat.c
>>> [0]PETSC ERROR: #4 PetscRegisterFinalizeAll() line 389 in
>>> /Users/markadams/Codes/petsc/src/sys/objects/destroy.c