[petsc-dev] Is master broken?

[Mark Adams via petsc-dev]

I am seeing this when I pull master into my branch:

"/autofs/nccs-svm1_home1/adams/petsc/src/mat/impls/dense/seq/cuda/
densecuda.cu"
  , line 243: error: function call is not allowed in a constant
  expression
  #if PETSC_PKG_CUDA_VERSION_GE(10,1,0)

and I see that this macro does not seem to be defined:

22:24 master= ~/Codes/petsc$ git grep PETSC_PKG_CUDA_VERSION_GE
src/mat/impls/dense/seq/cuda/densecuda.cu:#if
PETSC_PKG_CUDA_VERSION_GE(10,1,0)

Re: [petsc-dev] Is master broken?

[Smith, Barry F. via petsc-dev]

  It is generated automatically and put in 
arch-linux2-c-debug/include/petscpkg_version.h  this include file is included 
at top of the "bad" source  file crashes so in theory everything is in order 
check that arch-linux2-c-debug/include/petscpkg_version.h contains 
PETSC_PKG_CUDA_VERSION_GE and similar macros. If not send configure.lo

check what is in arch-linux2-c-debug/include/petscpkg_version.h it nothing or 
broken send configure.lo


  Barry



> On Jul 31, 2019, at 9:28 PM, Mark Adams via petsc-dev  
> wrote:
> 
> I am seeing this when I pull master into my branch:
> 
> "/autofs/nccs-svm1_home1/adams/petsc/src/mat/impls/dense/seq/cuda/densecuda.cu"
>   , line 243: error: function call is not allowed in a constant
>   expression
>   #if PETSC_PKG_CUDA_VERSION_GE(10,1,0)
> 
> and I see that this macro does not seem to be defined:
> 
> 22:24 master= ~/Codes/petsc$ git grep PETSC_PKG_CUDA_VERSION_GE
> src/mat/impls/dense/seq/cuda/densecuda.cu:#if 
> PETSC_PKG_CUDA_VERSION_GE(10,1,0)

Re: [petsc-dev] Is master broken?

[Mark Adams via petsc-dev]

I get the same error with a fresh clone of master.

On Thu, Aug 1, 2019 at 6:03 AM Mark Adams  wrote:

> Tried again after deleting the arch dirs and still have it.
> This is my branch that just merged master. I will try with just master.
> Thanks,
>
> On Thu, Aug 1, 2019 at 1:36 AM Smith, Barry F.  wrote:
>
>>
>>   It is generated automatically and put in
>> arch-linux2-c-debug/include/petscpkg_version.h  this include file is
>> included at top of the "bad" source  file crashes so in theory everything
>> is in order check that arch-linux2-c-debug/include/petscpkg_version.h
>> contains PETSC_PKG_CUDA_VERSION_GE and similar macros. If not send
>> configure.lo
>>
>> check what is in arch-linux2-c-debug/include/petscpkg_version.h it
>> nothing or broken send configure.lo
>>
>>
>>   Barry
>>
>>
>>
>> > On Jul 31, 2019, at 9:28 PM, Mark Adams via petsc-dev <
>> petsc-dev@mcs.anl.gov> wrote:
>> >
>> > I am seeing this when I pull master into my branch:
>> >
>> > "/autofs/nccs-svm1_home1/adams/petsc/src/mat/impls/dense/seq/cuda/
>> densecuda.cu"
>> >   , line 243: error: function call is not allowed in a constant
>> >   expression
>> >   #if PETSC_PKG_CUDA_VERSION_GE(10,1,0)
>> >
>> > and I see that this macro does not seem to be defined:
>> >
>> > 22:24 master= ~/Codes/petsc$ git grep PETSC_PKG_CUDA_VERSION_GE
>> > src/mat/impls/dense/seq/cuda/densecuda.cu:#if
>> PETSC_PKG_CUDA_VERSION_GE(10,1,0)
>>
>>

Re: [petsc-dev] Is master broken?

[Smith, Barry F. via petsc-dev]

Re: [petsc-dev] Is master broken?

[Mark Adams via petsc-dev]

On Thu, Aug 1, 2019 at 10:30 AM Smith, Barry F.  wrote:

>
>   Send
>
> ls arch-linux2-c-debug/include/
>

That is not my arch name. It is something like arch-summit-dbg64-pgi-cuda

>
>  arch-linux2-c-debug/include/petscpkg_version.h
>
> and configure.log
>
>
>
> > On Aug 1, 2019, at 5:23 AM, Mark Adams  wrote:
> >
> > I get the same error with a fresh clone of master.
> >
> > On Thu, Aug 1, 2019 at 6:03 AM Mark Adams  wrote:
> > Tried again after deleting the arch dirs and still have it.
> > This is my branch that just merged master. I will try with just master.
> > Thanks,
> >
> > On Thu, Aug 1, 2019 at 1:36 AM Smith, Barry F. 
> wrote:
> >
> >   It is generated automatically and put in
> arch-linux2-c-debug/include/petscpkg_version.h  this include file is
> included at top of the "bad" source  file crashes so in theory everything
> is in order check that arch-linux2-c-debug/include/petscpkg_version.h
> contains PETSC_PKG_CUDA_VERSION_GE and similar macros. If not send
> configure.lo
> >
> > check what is in arch-linux2-c-debug/include/petscpkg_version.h it
> nothing or broken send configure.lo
> >
> >
> >   Barry
> >
> >
> >
> > > On Jul 31, 2019, at 9:28 PM, Mark Adams via petsc-dev <
> petsc-dev@mcs.anl.gov> wrote:
> > >
> > > I am seeing this when I pull master into my branch:
> > >
> > > "/autofs/nccs-svm1_home1/adams/petsc/src/mat/impls/dense/seq/cuda/
> densecuda.cu"
> > >   , line 243: error: function call is not allowed in a constant
> > >   expression
> > >   #if PETSC_PKG_CUDA_VERSION_GE(10,1,0)
> > >
> > > and I see that this macro does not seem to be defined:
> > >
> > > 22:24 master= ~/Codes/petsc$ git grep PETSC_PKG_CUDA_VERSION_GE
> > > src/mat/impls/dense/seq/cuda/densecuda.cu:#if
> PETSC_PKG_CUDA_VERSION_GE(10,1,0)
> >
>
>

Re: [petsc-dev] Is master broken?

[Smith, Barry F. via petsc-dev]

  Unrelated question: Surely on a 30 million dollar machine it can do better 
than --download-fblaslapack ?

> On Aug 1, 2019, at 11:18 AM, Mark Adams  wrote:
> 
> 12:10 master= ~/petsc-master$ ls arch-summit-dbg64-pgi-cuda
> bin  externalpackages  include  lib  obj  share  tests
> 
> 12:11 master= ~/petsc-master$ tar cvfz ../tt.tarz 
> arch-summit-dbg64-pgi-cuda/include/petscpkg_version.h arch-summure.log
> arch-summit-dbg64-pgi-cuda/include/petscpkg_version.h
> arch-summit-dbg64-pgi-cuda/lib/petsc/conf/configure.log
> 
> 
> 
> 
> On Thu, Aug 1, 2019 at 12:08 PM Mark Adams  wrote:
> 
> 
> On Thu, Aug 1, 2019 at 10:30 AM Smith, Barry F.  wrote:
> 
>   Send
> 
> ls arch-linux2-c-debug/include/
> 
> That is not my arch name. It is something like arch-summit-dbg64-pgi-cuda 
> 
>  arch-linux2-c-debug/include/petscpkg_version.h
> 
> and configure.log
> 
> 
> 
> > On Aug 1, 2019, at 5:23 AM, Mark Adams  wrote:
> > 
> > I get the same error with a fresh clone of master.
> > 
> > On Thu, Aug 1, 2019 at 6:03 AM Mark Adams  wrote:
> > Tried again after deleting the arch dirs and still have it.
> > This is my branch that just merged master. I will try with just master.
> > Thanks,
> > 
> > On Thu, Aug 1, 2019 at 1:36 AM Smith, Barry F.  wrote:
> > 
> >   It is generated automatically and put in 
> > arch-linux2-c-debug/include/petscpkg_version.h  this include file is 
> > included at top of the "bad" source  file crashes so in theory everything 
> > is in order check that arch-linux2-c-debug/include/petscpkg_version.h 
> > contains PETSC_PKG_CUDA_VERSION_GE and similar macros. If not send 
> > configure.lo
> > 
> > check what is in arch-linux2-c-debug/include/petscpkg_version.h it nothing 
> > or broken send configure.lo
> > 
> > 
> >   Barry
> > 
> > 
> > 
> > > On Jul 31, 2019, at 9:28 PM, Mark Adams via petsc-dev 
> > >  wrote:
> > > 
> > > I am seeing this when I pull master into my branch:
> > > 
> > > "/autofs/nccs-svm1_home1/adams/petsc/src/mat/impls/dense/seq/cuda/densecuda.cu"
> > >   , line 243: error: function call is not allowed in a constant
> > >   expression
> > >   #if PETSC_PKG_CUDA_VERSION_GE(10,1,0)
> > > 
> > > and I see that this macro does not seem to be defined:
> > > 
> > > 22:24 master= ~/Codes/petsc$ git grep PETSC_PKG_CUDA_VERSION_GE
> > > src/mat/impls/dense/seq/cuda/densecuda.cu:#if 
> > > PETSC_PKG_CUDA_VERSION_GE(10,1,0)
> > 
> 
> 

Re: [petsc-dev] Is master broken?

[Smith, Barry F. via petsc-dev]

  Please run with the branch barry/2019-09-01/robustify-version-check and send 
configure.log


> On Aug 1, 2019, at 11:18 AM, Mark Adams  wrote:
> 
> 12:10 master= ~/petsc-master$ ls arch-summit-dbg64-pgi-cuda
> bin  externalpackages  include  lib  obj  share  tests
> 
> 12:11 master= ~/petsc-master$ tar cvfz ../tt.tarz 
> arch-summit-dbg64-pgi-cuda/include/petscpkg_version.h arch-summure.log
> arch-summit-dbg64-pgi-cuda/include/petscpkg_version.h
> arch-summit-dbg64-pgi-cuda/lib/petsc/conf/configure.log
> 
> 
> 
> 
> On Thu, Aug 1, 2019 at 12:08 PM Mark Adams  wrote:
> 
> 
> On Thu, Aug 1, 2019 at 10:30 AM Smith, Barry F.  wrote:
> 
>   Send
> 
> ls arch-linux2-c-debug/include/
> 
> That is not my arch name. It is something like arch-summit-dbg64-pgi-cuda 
> 
>  arch-linux2-c-debug/include/petscpkg_version.h
> 
> and configure.log
> 
> 
> 
> > On Aug 1, 2019, at 5:23 AM, Mark Adams  wrote:
> > 
> > I get the same error with a fresh clone of master.
> > 
> > On Thu, Aug 1, 2019 at 6:03 AM Mark Adams  wrote:
> > Tried again after deleting the arch dirs and still have it.
> > This is my branch that just merged master. I will try with just master.
> > Thanks,
> > 
> > On Thu, Aug 1, 2019 at 1:36 AM Smith, Barry F.  wrote:
> > 
> >   It is generated automatically and put in 
> > arch-linux2-c-debug/include/petscpkg_version.h  this include file is 
> > included at top of the "bad" source  file crashes so in theory everything 
> > is in order check that arch-linux2-c-debug/include/petscpkg_version.h 
> > contains PETSC_PKG_CUDA_VERSION_GE and similar macros. If not send 
> > configure.lo
> > 
> > check what is in arch-linux2-c-debug/include/petscpkg_version.h it nothing 
> > or broken send configure.lo
> > 
> > 
> >   Barry
> > 
> > 
> > 
> > > On Jul 31, 2019, at 9:28 PM, Mark Adams via petsc-dev 
> > >  wrote:
> > > 
> > > I am seeing this when I pull master into my branch:
> > > 
> > > "/autofs/nccs-svm1_home1/adams/petsc/src/mat/impls/dense/seq/cuda/densecuda.cu"
> > >   , line 243: error: function call is not allowed in a constant
> > >   expression
> > >   #if PETSC_PKG_CUDA_VERSION_GE(10,1,0)
> > > 
> > > and I see that this macro does not seem to be defined:
> > > 
> > > 22:24 master= ~/Codes/petsc$ git grep PETSC_PKG_CUDA_VERSION_GE
> > > src/mat/impls/dense/seq/cuda/densecuda.cu:#if 
> > > PETSC_PKG_CUDA_VERSION_GE(10,1,0)
> > 
> 
> 

Re: [petsc-dev] Is master broken?

[Smith, Barry F. via petsc-dev]

  Mark,

Thanks, that was not expected to work, I was just verifying the exact cause 
of the problem and it was what I was guessing.

I believe I have fixed it. Please pull that branch again and let me know if 
it works. If it does we'll do rush testing and get it into master.

 Thanks

 Barry


> On Aug 1, 2019, at 11:08 AM, Mark Adams  wrote:
> 
> 
> 
> On Thu, Aug 1, 2019 at 10:30 AM Smith, Barry F.  wrote:
> 
>   Send
> 
> ls arch-linux2-c-debug/include/
> 
> That is not my arch name. It is something like arch-summit-dbg64-pgi-cuda 
> 
>  arch-linux2-c-debug/include/petscpkg_version.h
> 
> and configure.log
> 
> 
> 
> > On Aug 1, 2019, at 5:23 AM, Mark Adams  wrote:
> > 
> > I get the same error with a fresh clone of master.
> > 
> > On Thu, Aug 1, 2019 at 6:03 AM Mark Adams  wrote:
> > Tried again after deleting the arch dirs and still have it.
> > This is my branch that just merged master. I will try with just master.
> > Thanks,
> > 
> > On Thu, Aug 1, 2019 at 1:36 AM Smith, Barry F.  wrote:
> > 
> >   It is generated automatically and put in 
> > arch-linux2-c-debug/include/petscpkg_version.h  this include file is 
> > included at top of the "bad" source  file crashes so in theory everything 
> > is in order check that arch-linux2-c-debug/include/petscpkg_version.h 
> > contains PETSC_PKG_CUDA_VERSION_GE and similar macros. If not send 
> > configure.lo
> > 
> > check what is in arch-linux2-c-debug/include/petscpkg_version.h it nothing 
> > or broken send configure.lo
> > 
> > 
> >   Barry
> > 
> > 
> > 
> > > On Jul 31, 2019, at 9:28 PM, Mark Adams via petsc-dev 
> > >  wrote:
> > > 
> > > I am seeing this when I pull master into my branch:
> > > 
> > > "/autofs/nccs-svm1_home1/adams/petsc/src/mat/impls/dense/seq/cuda/densecuda.cu"
> > >   , line 243: error: function call is not allowed in a constant
> > >   expression
> > >   #if PETSC_PKG_CUDA_VERSION_GE(10,1,0)
> > > 
> > > and I see that this macro does not seem to be defined:
> > > 
> > > 22:24 master= ~/Codes/petsc$ git grep PETSC_PKG_CUDA_VERSION_GE
> > > src/mat/impls/dense/seq/cuda/densecuda.cu:#if 
> > > PETSC_PKG_CUDA_VERSION_GE(10,1,0)
> > 
> 

Re: [petsc-dev] Is master broken?

[Smith, Barry F. via petsc-dev]

  Yes, these are bugs in Stefano's work that got into master because we didn't 
have comprehensive testing. There are two branches in the PR list you can 
cherry pick that will fix this problem. Sorry about this. We're trying to get 
them into master as quickly as possible but 

   Barry


> On Aug 2, 2019, at 8:39 AM, Mark Adams  wrote:
> 
> closer,
> 
> On Fri, Aug 2, 2019 at 9:13 AM Smith, Barry F.  wrote:
> 
>   Mark,
> 
> Thanks, that was not expected to work, I was just verifying the exact 
> cause of the problem and it was what I was guessing.
> 
> I believe I have fixed it. Please pull that branch again and let me know 
> if it works. If it does we'll do rush testing and get it into master.
> 
>  Thanks
> 
>  Barry
> 
> 
> > On Aug 1, 2019, at 11:08 AM, Mark Adams  wrote:
> > 
> > 
> > 
> > On Thu, Aug 1, 2019 at 10:30 AM Smith, Barry F.  wrote:
> > 
> >   Send
> > 
> > ls arch-linux2-c-debug/include/
> > 
> > That is not my arch name. It is something like arch-summit-dbg64-pgi-cuda 
> > 
> >  arch-linux2-c-debug/include/petscpkg_version.h
> > 
> > and configure.log
> > 
> > 
> > 
> > > On Aug 1, 2019, at 5:23 AM, Mark Adams  wrote:
> > > 
> > > I get the same error with a fresh clone of master.
> > > 
> > > On Thu, Aug 1, 2019 at 6:03 AM Mark Adams  wrote:
> > > Tried again after deleting the arch dirs and still have it.
> > > This is my branch that just merged master. I will try with just master.
> > > Thanks,
> > > 
> > > On Thu, Aug 1, 2019 at 1:36 AM Smith, Barry F.  wrote:
> > > 
> > >   It is generated automatically and put in 
> > > arch-linux2-c-debug/include/petscpkg_version.h  this include file is 
> > > included at top of the "bad" source  file crashes so in theory everything 
> > > is in order check that arch-linux2-c-debug/include/petscpkg_version.h 
> > > contains PETSC_PKG_CUDA_VERSION_GE and similar macros. If not send 
> > > configure.lo
> > > 
> > > check what is in arch-linux2-c-debug/include/petscpkg_version.h it 
> > > nothing or broken send configure.lo
> > > 
> > > 
> > >   Barry
> > > 
> > > 
> > > 
> > > > On Jul 31, 2019, at 9:28 PM, Mark Adams via petsc-dev 
> > > >  wrote:
> > > > 
> > > > I am seeing this when I pull master into my branch:
> > > > 
> > > > "/autofs/nccs-svm1_home1/adams/petsc/src/mat/impls/dense/seq/cuda/densecuda.cu"
> > > >   , line 243: error: function call is not allowed in a constant
> > > >   expression
> > > >   #if PETSC_PKG_CUDA_VERSION_GE(10,1,0)
> > > > 
> > > > and I see that this macro does not seem to be defined:
> > > > 
> > > > 22:24 master= ~/Codes/petsc$ git grep PETSC_PKG_CUDA_VERSION_GE
> > > > src/mat/impls/dense/seq/cuda/densecuda.cu:#if 
> > > > PETSC_PKG_CUDA_VERSION_GE(10,1,0)
> > > 
> > 
> 
> 

Re: [petsc-dev] Is master broken?

[Karl Rupp via petsc-dev]

FYI: The two branches are currently testing in `next-tmp` and are likely 
to be merged to master in ~5 hours.


Best regards,
Karli


On 8/2/19 4:53 PM, Smith, Barry F. via petsc-dev wrote:


   Yes, these are bugs in Stefano's work that got into master because we didn't 
have comprehensive testing. There are two branches in the PR list you can 
cherry pick that will fix this problem. Sorry about this. We're trying to get 
them into master as quickly as possible but 

Barry



On Aug 2, 2019, at 8:39 AM, Mark Adams  wrote:

closer,

On Fri, Aug 2, 2019 at 9:13 AM Smith, Barry F.  wrote:

   Mark,

 Thanks, that was not expected to work, I was just verifying the exact 
cause of the problem and it was what I was guessing.

 I believe I have fixed it. Please pull that branch again and let me know 
if it works. If it does we'll do rush testing and get it into master.

  Thanks

  Barry



On Aug 1, 2019, at 11:08 AM, Mark Adams  wrote:



On Thu, Aug 1, 2019 at 10:30 AM Smith, Barry F.  wrote:

   Send

ls arch-linux2-c-debug/include/

That is not my arch name. It is something like arch-summit-dbg64-pgi-cuda

  arch-linux2-c-debug/include/petscpkg_version.h

and configure.log




On Aug 1, 2019, at 5:23 AM, Mark Adams  wrote:

I get the same error with a fresh clone of master.

On Thu, Aug 1, 2019 at 6:03 AM Mark Adams  wrote:
Tried again after deleting the arch dirs and still have it.
This is my branch that just merged master. I will try with just master.
Thanks,

On Thu, Aug 1, 2019 at 1:36 AM Smith, Barry F.  wrote:

   It is generated automatically and put in 
arch-linux2-c-debug/include/petscpkg_version.h  this include file is included at top of 
the "bad" source  file crashes so in theory everything is in order check that 
arch-linux2-c-debug/include/petscpkg_version.h contains PETSC_PKG_CUDA_VERSION_GE and 
similar macros. If not send configure.lo

check what is in arch-linux2-c-debug/include/petscpkg_version.h it nothing or 
broken send configure.lo


   Barry




On Jul 31, 2019, at 9:28 PM, Mark Adams via petsc-dev  
wrote:

I am seeing this when I pull master into my branch:

"/autofs/nccs-svm1_home1/adams/petsc/src/mat/impls/dense/seq/cuda/densecuda.cu"
   , line 243: error: function call is not allowed in a constant
   expression
   #if PETSC_PKG_CUDA_VERSION_GE(10,1,0)

and I see that this macro does not seem to be defined:

22:24 master= ~/Codes/petsc$ git grep PETSC_PKG_CUDA_VERSION_GE
src/mat/impls/dense/seq/cuda/densecuda.cu:#if PETSC_PKG_CUDA_VERSION_GE(10,1,0)

Re: [petsc-dev] Is master broken?

[Mark Adams via petsc-dev]

I picked these two into Barry's branch and it built.

I would like to get them into my cuda branch. Should I just pick them? And
not worry about Barry's branch. Or will that not work.

On Fri, Aug 2, 2019 at 12:03 PM Karl Rupp  wrote:

> FYI: The two branches are currently testing in `next-tmp` and are likely
> to be merged to master in ~5 hours.
>
> Best regards,
> Karli
>
>
> On 8/2/19 4:53 PM, Smith, Barry F. via petsc-dev wrote:
> >
> >Yes, these are bugs in Stefano's work that got into master because we
> didn't have comprehensive testing. There are two branches in the PR list
> you can cherry pick that will fix this problem. Sorry about this. We're
> trying to get them into master as quickly as possible but 
> >
> > Barry
> >
> >
> >> On Aug 2, 2019, at 8:39 AM, Mark Adams  wrote:
> >>
> >> closer,
> >>
> >> On Fri, Aug 2, 2019 at 9:13 AM Smith, Barry F. 
> wrote:
> >>
> >>Mark,
> >>
> >>  Thanks, that was not expected to work, I was just verifying the
> exact cause of the problem and it was what I was guessing.
> >>
> >>  I believe I have fixed it. Please pull that branch again and let
> me know if it works. If it does we'll do rush testing and get it into
> master.
> >>
> >>   Thanks
> >>
> >>   Barry
> >>
> >>
> >>> On Aug 1, 2019, at 11:08 AM, Mark Adams  wrote:
> >>>
> >>>
> >>>
> >>> On Thu, Aug 1, 2019 at 10:30 AM Smith, Barry F. 
> wrote:
> >>>
> >>>Send
> >>>
> >>> ls arch-linux2-c-debug/include/
> >>>
> >>> That is not my arch name. It is something like
> arch-summit-dbg64-pgi-cuda
> >>>
> >>>   arch-linux2-c-debug/include/petscpkg_version.h
> >>>
> >>> and configure.log
> >>>
> >>>
> >>>
>  On Aug 1, 2019, at 5:23 AM, Mark Adams  wrote:
> 
>  I get the same error with a fresh clone of master.
> 
>  On Thu, Aug 1, 2019 at 6:03 AM Mark Adams  wrote:
>  Tried again after deleting the arch dirs and still have it.
>  This is my branch that just merged master. I will try with just
> master.
>  Thanks,
> 
>  On Thu, Aug 1, 2019 at 1:36 AM Smith, Barry F. 
> wrote:
> 
> It is generated automatically and put in
> arch-linux2-c-debug/include/petscpkg_version.h  this include file is
> included at top of the "bad" source  file crashes so in theory everything
> is in order check that arch-linux2-c-debug/include/petscpkg_version.h
> contains PETSC_PKG_CUDA_VERSION_GE and similar macros. If not send
> configure.lo
> 
>  check what is in arch-linux2-c-debug/include/petscpkg_version.h it
> nothing or broken send configure.lo
> 
> 
> Barry
> 
> 
> 
> > On Jul 31, 2019, at 9:28 PM, Mark Adams via petsc-dev <
> petsc-dev@mcs.anl.gov> wrote:
> >
> > I am seeing this when I pull master into my branch:
> >
> > "/autofs/nccs-svm1_home1/adams/petsc/src/mat/impls/dense/seq/cuda/
> densecuda.cu"
> >, line 243: error: function call is not allowed in a
> constant
> >expression
> >#if PETSC_PKG_CUDA_VERSION_GE(10,1,0)
> >
> > and I see that this macro does not seem to be defined:
> >
> > 22:24 master= ~/Codes/petsc$ git grep PETSC_PKG_CUDA_VERSION_GE
> > src/mat/impls/dense/seq/cuda/densecuda.cu:#if
> PETSC_PKG_CUDA_VERSION_GE(10,1,0)
> 
> >>>
> >>
> >> 
> >
>

Re: [petsc-dev] Is master broken?

[Karl Rupp via petsc-dev]

You should be able to just cherry-pick the commits from Barry's branch 
as well as the two other branches.




On 8/2/19 8:13 PM, Mark Adams wrote:

I picked these two into Barry's branch and it built.

I would like to get them into my cuda branch. Should I just pick them? 
And not worry about Barry's branch. Or will that not work.


On Fri, Aug 2, 2019 at 12:03 PM Karl Rupp > wrote:


FYI: The two branches are currently testing in `next-tmp` and are
likely
to be merged to master in ~5 hours.

Best regards,
Karli


On 8/2/19 4:53 PM, Smith, Barry F. via petsc-dev wrote:
 >
 >    Yes, these are bugs in Stefano's work that got into master
because we didn't have comprehensive testing. There are two branches
in the PR list you can cherry pick that will fix this problem. Sorry
about this. We're trying to get them into master as quickly as
possible but 
 >
 >     Barry
 >
 >
 >> On Aug 2, 2019, at 8:39 AM, Mark Adams mailto:mfad...@lbl.gov>> wrote:
 >>
 >> closer,
 >>
 >> On Fri, Aug 2, 2019 at 9:13 AM Smith, Barry F.
mailto:bsm...@mcs.anl.gov>> wrote:
 >>
 >>    Mark,
 >>
 >>      Thanks, that was not expected to work, I was just verifying
the exact cause of the problem and it was what I was guessing.
 >>
 >>      I believe I have fixed it. Please pull that branch again
and let me know if it works. If it does we'll do rush testing and
get it into master.
 >>
 >>       Thanks
 >>
 >>       Barry
 >>
 >>
 >>> On Aug 1, 2019, at 11:08 AM, Mark Adams mailto:mfad...@lbl.gov>> wrote:
 >>>
 >>>
 >>>
 >>> On Thu, Aug 1, 2019 at 10:30 AM Smith, Barry F.
mailto:bsm...@mcs.anl.gov>> wrote:
 >>>
 >>>    Send
 >>>
 >>> ls arch-linux2-c-debug/include/
 >>>
 >>> That is not my arch name. It is something like
arch-summit-dbg64-pgi-cuda
 >>>
 >>>   arch-linux2-c-debug/include/petscpkg_version.h
 >>>
 >>> and configure.log
 >>>
 >>>
 >>>
  On Aug 1, 2019, at 5:23 AM, Mark Adams mailto:mfad...@lbl.gov>> wrote:
 
  I get the same error with a fresh clone of master.
 
  On Thu, Aug 1, 2019 at 6:03 AM Mark Adams mailto:mfad...@lbl.gov>> wrote:
  Tried again after deleting the arch dirs and still have it.
  This is my branch that just merged master. I will try with
just master.
  Thanks,
 
  On Thu, Aug 1, 2019 at 1:36 AM Smith, Barry F.
mailto:bsm...@mcs.anl.gov>> wrote:
 
     It is generated automatically and put in
arch-linux2-c-debug/include/petscpkg_version.h  this include file is
included at top of the "bad" source  file crashes so in theory
everything is in order check that
arch-linux2-c-debug/include/petscpkg_version.h contains
PETSC_PKG_CUDA_VERSION_GE and similar macros. If not send configure.lo
 
  check what is in
arch-linux2-c-debug/include/petscpkg_version.h it nothing or broken
send configure.lo
 
 
     Barry
 
 
 
 > On Jul 31, 2019, at 9:28 PM, Mark Adams via petsc-dev
mailto:petsc-dev@mcs.anl.gov>> wrote:
 >
 > I am seeing this when I pull master into my branch:
 >
 >

"/autofs/nccs-svm1_home1/adams/petsc/src/mat/impls/dense/seq/cuda/densecuda.cu
"
 >            , line 243: error: function call is not allowed in
a constant
 >            expression
 >    #if PETSC_PKG_CUDA_VERSION_GE(10,1,0)
 >
 > and I see that this macro does not seem to be defined:
 >
 > 22:24 master= ~/Codes/petsc$ git grep PETSC_PKG_CUDA_VERSION_GE
 > src/mat/impls/dense/seq/cuda/densecuda.cu:#if
PETSC_PKG_CUDA_VERSION_GE(10,1,0)
 
 >>>
 >>
 >> 
 >

Re: [petsc-dev] Is master broken?

[Mark Adams via petsc-dev]

I have been cherry-picking, etc, branch mark/gamg-fix-viennacl-rebased and
it is very messed up. Can someone please update this branch when all the
fixes are settled down? eg, I am seeing dozens of modified files that I
don't know anything about and I certainly don't want to put in a PR for
them.

I also seem to lose my pinToCPU method for cuda matrices. I don't
understand how that conflicted with anyone else but it did.

Thanks,
Mark

Re: [petsc-dev] Is master broken?

[Balay, Satish via petsc-dev]

I've attempted to rebase this branch over latest master - and pushed
my changes to branch mark/gamg-fix-viennacl-rebased-v2

You might want to check each of your commits in this branch to see if
they are ok. I had to add one extra commit - to make it match 'merge
of mark/gamg-fix-viennacl-rebased and master'.

This branch has 21 commits. I think its best if you can collapse them
into reasonable chunks of changes. [presumably a single commit for all
the changes is not the correct thing here. But the current set of 21
commits are all over the place]

If you are able to migrate to this branch - its best to delete the old
one [i.e origin/mark/gamg-fix-viennacl-rebased]

Satish

On Fri, 2 Aug 2019, Mark Adams via petsc-dev wrote:

> I have been cherry-picking, etc, branch mark/gamg-fix-viennacl-rebased and
> it is very messed up. Can someone please update this branch when all the
> fixes are settled down? eg, I am seeing dozens of modified files that I
> don't know anything about and I certainly don't want to put in a PR for
> them.
> 
> I also seem to lose my pinToCPU method for cuda matrices. I don't
> understand how that conflicted with anyone else but it did.
> 
> Thanks,
> Mark
> 

Re: [petsc-dev] Is master broken?

[Karl Rupp via petsc-dev]

If you ignore the initial ViennaCL-related commits and check against 
current master (that just received cherry-picked updates from your PR), 
then there are really only a few commits left that are not yet integrated.


(I'll extract two more PRs on Monday, so master will soon have your 
fixes in.)


Best regards,
Karli


On 8/3/19 5:21 AM, Balay, Satish wrote:

I've attempted to rebase this branch over latest master - and pushed
my changes to branch mark/gamg-fix-viennacl-rebased-v2

You might want to check each of your commits in this branch to see if
they are ok. I had to add one extra commit - to make it match 'merge
of mark/gamg-fix-viennacl-rebased and master'.

This branch has 21 commits. I think its best if you can collapse them
into reasonable chunks of changes. [presumably a single commit for all
the changes is not the correct thing here. But the current set of 21
commits are all over the place]

If you are able to migrate to this branch - its best to delete the old
one [i.e origin/mark/gamg-fix-viennacl-rebased]

Satish

On Fri, 2 Aug 2019, Mark Adams via petsc-dev wrote:


I have been cherry-picking, etc, branch mark/gamg-fix-viennacl-rebased and
it is very messed up. Can someone please update this branch when all the
fixes are settled down? eg, I am seeing dozens of modified files that I
don't know anything about and I certainly don't want to put in a PR for
them.

I also seem to lose my pinToCPU method for cuda matrices. I don't
understand how that conflicted with anyone else but it did.

Thanks,
Mark

Re: [petsc-dev] Is master broken?

[Mark Adams via petsc-dev]

Karl,
Did you want me to do anything at this point? (on vacation this week) I
will verify that master is all fixed if you get all my stuff integrated
when I get back to work in a week.
Thanks,
Mark

On Sat, Aug 3, 2019 at 10:50 AM Karl Rupp  wrote:

> If you ignore the initial ViennaCL-related commits and check against
> current master (that just received cherry-picked updates from your PR),
> then there are really only a few commits left that are not yet integrated.
>
> (I'll extract two more PRs on Monday, so master will soon have your
> fixes in.)
>
> Best regards,
> Karli
>
>
> On 8/3/19 5:21 AM, Balay, Satish wrote:
> > I've attempted to rebase this branch over latest master - and pushed
> > my changes to branch mark/gamg-fix-viennacl-rebased-v2
> >
> > You might want to check each of your commits in this branch to see if
> > they are ok. I had to add one extra commit - to make it match 'merge
> > of mark/gamg-fix-viennacl-rebased and master'.
> >
> > This branch has 21 commits. I think its best if you can collapse them
> > into reasonable chunks of changes. [presumably a single commit for all
> > the changes is not the correct thing here. But the current set of 21
> > commits are all over the place]
> >
> > If you are able to migrate to this branch - its best to delete the old
> > one [i.e origin/mark/gamg-fix-viennacl-rebased]
> >
> > Satish
> >
> > On Fri, 2 Aug 2019, Mark Adams via petsc-dev wrote:
> >
> >> I have been cherry-picking, etc, branch mark/gamg-fix-viennacl-rebased
> and
> >> it is very messed up. Can someone please update this branch when all the
> >> fixes are settled down? eg, I am seeing dozens of modified files that I
> >> don't know anything about and I certainly don't want to put in a PR for
> >> them.
> >>
> >> I also seem to lose my pinToCPU method for cuda matrices. I don't
> >> understand how that conflicted with anyone else but it did.
> >>
> >> Thanks,
> >> Mark
> >>
> >
>

Re: [petsc-dev] Is master broken?

[Karl Rupp via petsc-dev]

Hi Mark,

it's fine if you just double-check that all your fixes are in master 
when you're back :-)


Best regards and enjoy your vacation,
Karli


On 8/3/19 8:47 PM, Mark Adams wrote:

Karl,
Did you want me to do anything at this point? (on vacation this week) I 
will verify that master is all fixed if you get all my stuff integrated 
when I get back to work in a week.

Thanks,
Mark

On Sat, Aug 3, 2019 at 10:50 AM Karl Rupp > wrote:


If you ignore the initial ViennaCL-related commits and check against
current master (that just received cherry-picked updates from your PR),
then there are really only a few commits left that are not yet
integrated.

(I'll extract two more PRs on Monday, so master will soon have your
fixes in.)

Best regards,
Karli


On 8/3/19 5:21 AM, Balay, Satish wrote:
 > I've attempted to rebase this branch over latest master - and pushed
 > my changes to branch mark/gamg-fix-viennacl-rebased-v2
 >
 > You might want to check each of your commits in this branch to see if
 > they are ok. I had to add one extra commit - to make it match 'merge
 > of mark/gamg-fix-viennacl-rebased and master'.
 >
 > This branch has 21 commits. I think its best if you can collapse them
 > into reasonable chunks of changes. [presumably a single commit
for all
 > the changes is not the correct thing here. But the current set of 21
 > commits are all over the place]
 >
 > If you are able to migrate to this branch - its best to delete
the old
 > one [i.e origin/mark/gamg-fix-viennacl-rebased]
 >
 > Satish
 >
 > On Fri, 2 Aug 2019, Mark Adams via petsc-dev wrote:
 >
 >> I have been cherry-picking, etc, branch
mark/gamg-fix-viennacl-rebased and
 >> it is very messed up. Can someone please update this branch when
all the
 >> fixes are settled down? eg, I am seeing dozens of modified files
that I
 >> don't know anything about and I certainly don't want to put in a
PR for
 >> them.
 >>
 >> I also seem to lose my pinToCPU method for cuda matrices. I don't
 >> understand how that conflicted with anyone else but it did.
 >>
 >> Thanks,
 >> Mark
 >>
 >

Re: [petsc-dev] Is master broken?

[Mark Adams via petsc-dev]

Satish,

Your new branch mark/gamg-fix-viennacl-rebased-v2 does not seem to have
Barry's fixes (the source of this thread):

 ...
  , line 243: error: function call is not allowed in a constant
  expression
  #if PETSC_PKG_CUDA_VERSION_GE(10,1,0)

Here is the reflog of the cherry picking that I did to fix my last version.
I forget exactly what I did to get these changes so I'd like to not mess it
up. Can you add these to your new branch?

Thanks,
Mark

06:10 2 mark/gamg-fix-viennacl-rebased-v2= ~/petsc$ git reflog
3baf678 HEAD@{0}: checkout: moving from mark/gamg-fix-viennacl-rebased to
mark/gamg-fix-viennacl-rebased-v2
e50f779 HEAD@{1}: cherry-pick: 1) When detecting version info handle blanks
introducted by preprocessor, error if needed version cannot be det
87beba0 HEAD@{2}: cherry-pick: Use outputPreprocess instead of preprocess
since it prints source to log
9163512 HEAD@{3}: cherry-pick: Fix manual pages for
MatXXXYBAIJSetPreallocationCSR() routines
b4500af HEAD@{4}: cherry-pick: fix compile warnings on
arch-linux-pkgs-64idx triggered by c73702f59ac80eb68c197b7eea6d8474b9e9853c
9f418a9 HEAD@{5}: cherry-pick: Fix compile warnings --with-cuda
b541e45 HEAD@{6}: commit (cherry-pick): added a file that seemed to get
deleted
5d2c71f HEAD@{7}: checkout: moving from
barry/2019-09-01/robustify-version-check to mark/gamg-fix-viennacl-rebased
7c2e96e HEAD@{8}: cherry-pick: Fix compile warnings --with-cuda
d70ea55 HEAD@{9}: cherry-pick: MATSEQDENSECUDA: Fix warnings
c85f03d HEAD@{10}: pull origin: Fast-forward
e24ebd8 HEAD@{11}: checkout: moving from master to
barry/2019-09-01/robustify-version-check

>
>

Re: [petsc-dev] Is master broken?

[Mark Adams via petsc-dev]

Satish, I think I can do this now.
Mark

On Mon, Aug 12, 2019 at 6:26 AM Mark Adams  wrote:

> Satish,
>
> Your new branch mark/gamg-fix-viennacl-rebased-v2 does not seem to have
> Barry's fixes (the source of this thread):
>
>  ...
>   , line 243: error: function call is not allowed in a constant
>   expression
>   #if PETSC_PKG_CUDA_VERSION_GE(10,1,0)
>
> Here is the reflog of the cherry picking that I did to fix my last
> version. I forget exactly what I did to get these changes so I'd like to
> not mess it up. Can you add these to your new branch?
>
> Thanks,
> Mark
>
> 06:10 2 mark/gamg-fix-viennacl-rebased-v2= ~/petsc$ git reflog
> 3baf678 HEAD@{0}: checkout: moving from mark/gamg-fix-viennacl-rebased to
> mark/gamg-fix-viennacl-rebased-v2
> e50f779 HEAD@{1}: cherry-pick: 1) When detecting version info handle
> blanks introducted by preprocessor, error if needed version cannot be det
> 87beba0 HEAD@{2}: cherry-pick: Use outputPreprocess instead of preprocess
> since it prints source to log
> 9163512 HEAD@{3}: cherry-pick: Fix manual pages for
> MatXXXYBAIJSetPreallocationCSR() routines
> b4500af HEAD@{4}: cherry-pick: fix compile warnings on
> arch-linux-pkgs-64idx triggered by c73702f59ac80eb68c197b7eea6d8474b9e9853c
> 9f418a9 HEAD@{5}: cherry-pick: Fix compile warnings --with-cuda
> b541e45 HEAD@{6}: commit (cherry-pick): added a file that seemed to get
> deleted
> 5d2c71f HEAD@{7}: checkout: moving from
> barry/2019-09-01/robustify-version-check to mark/gamg-fix-viennacl-rebased
> 7c2e96e HEAD@{8}: cherry-pick: Fix compile warnings --with-cuda
> d70ea55 HEAD@{9}: cherry-pick: MATSEQDENSECUDA: Fix warnings
> c85f03d HEAD@{10}: pull origin: Fast-forward
> e24ebd8 HEAD@{11}: checkout: moving from master to
> barry/2019-09-01/robustify-version-check
>
>>
>>

Re: [petsc-dev] Is master broken?

[Karl Rupp via petsc-dev]

Hi Mark,

most of the CUDA-related fixes from your PR are now in master. Thank you!

The pinning of GPU-matrices to CPUs is not in master because it had 
several issues:


https://bitbucket.org/petsc/petsc/pull-requests/1954/cuda-fixes-to-pinning-onto-cpu/diff

The ViennaCL-related changes in mark/gamg-fix-viennacl-rebased can be 
safely discarded as the new GPU wrapper will come in place over the next 
days. ex56 has not been pulled over as it's not running properly on GPUs 
yet (the pinning in your branch effectively turned GPU matrices into 
normal PETSc matrices, effectively running (almost) everything on the 
CPU again)


So at this point I recommend to start a new branch off master and 
manually transfer over any bits from the pinning that you want to keep.


Best regards,
Karli


On 8/3/19 8:47 PM, Mark Adams wrote:

Karl,
Did you want me to do anything at this point? (on vacation this week) I 
will verify that master is all fixed if you get all my stuff integrated 
when I get back to work in a week.

Thanks,
Mark

On Sat, Aug 3, 2019 at 10:50 AM Karl Rupp > wrote:


If you ignore the initial ViennaCL-related commits and check against
current master (that just received cherry-picked updates from your PR),
then there are really only a few commits left that are not yet
integrated.

(I'll extract two more PRs on Monday, so master will soon have your
fixes in.)

Best regards,
Karli


On 8/3/19 5:21 AM, Balay, Satish wrote:
 > I've attempted to rebase this branch over latest master - and pushed
 > my changes to branch mark/gamg-fix-viennacl-rebased-v2
 >
 > You might want to check each of your commits in this branch to see if
 > they are ok. I had to add one extra commit - to make it match 'merge
 > of mark/gamg-fix-viennacl-rebased and master'.
 >
 > This branch has 21 commits. I think its best if you can collapse them
 > into reasonable chunks of changes. [presumably a single commit
for all
 > the changes is not the correct thing here. But the current set of 21
 > commits are all over the place]
 >
 > If you are able to migrate to this branch - its best to delete
the old
 > one [i.e origin/mark/gamg-fix-viennacl-rebased]
 >
 > Satish
 >
 > On Fri, 2 Aug 2019, Mark Adams via petsc-dev wrote:
 >
 >> I have been cherry-picking, etc, branch
mark/gamg-fix-viennacl-rebased and
 >> it is very messed up. Can someone please update this branch when
all the
 >> fixes are settled down? eg, I am seeing dozens of modified files
that I
 >> don't know anything about and I certainly don't want to put in a
PR for
 >> them.
 >>
 >> I also seem to lose my pinToCPU method for cuda matrices. I don't
 >> understand how that conflicted with anyone else but it did.
 >>
 >> Thanks,
 >> Mark
 >>
 >

Re: [petsc-dev] Is master broken?

[Mark Adams via petsc-dev]

On Mon, Aug 12, 2019 at 9:49 AM Karl Rupp  wrote:

> Hi Mark,
>
> most of the CUDA-related fixes from your PR are now in master. Thank you!
>
> The pinning of GPU-matrices to CPUs is not in master because it had
> several issues:
>
>
> https://bitbucket.org/petsc/petsc/pull-requests/1954/cuda-fixes-to-pinning-onto-cpu/diff
>
>
These links are dead.



> The ViennaCL-related changes in mark/gamg-fix-viennacl-rebased can be
> safely discarded as the new GPU wrapper will come in place over the next
> days. ex56 has not been pulled over as it's not running properly on GPUs
> yet (the pinning in your branch effectively turned GPU matrices into
> normal PETSc matrices, effectively running (almost) everything on the
> CPU again)
>
> So at this point I recommend to start a new branch off master and
> manually transfer over any bits from the pinning that you want to keep.
>

FYI, Satish worked on cleaning this branch up a week or two ago.


>
> Best regards,
> Karli
>
>
> On 8/3/19 8:47 PM, Mark Adams wrote:
> > Karl,
> > Did you want me to do anything at this point? (on vacation this week) I
> > will verify that master is all fixed if you get all my stuff integrated
> > when I get back to work in a week.
> > Thanks,
> > Mark
> >
> > On Sat, Aug 3, 2019 at 10:50 AM Karl Rupp  > > wrote:
> >
> > If you ignore the initial ViennaCL-related commits and check against
> > current master (that just received cherry-picked updates from your
> PR),
> > then there are really only a few commits left that are not yet
> > integrated.
> >
> > (I'll extract two more PRs on Monday, so master will soon have your
> > fixes in.)
> >
> > Best regards,
> > Karli
> >
> >
> > On 8/3/19 5:21 AM, Balay, Satish wrote:
> >  > I've attempted to rebase this branch over latest master - and
> pushed
> >  > my changes to branch mark/gamg-fix-viennacl-rebased-v2
> >  >
> >  > You might want to check each of your commits in this branch to
> see if
> >  > they are ok. I had to add one extra commit - to make it match
> 'merge
> >  > of mark/gamg-fix-viennacl-rebased and master'.
> >  >
> >  > This branch has 21 commits. I think its best if you can collapse
> them
> >  > into reasonable chunks of changes. [presumably a single commit
> > for all
> >  > the changes is not the correct thing here. But the current set of
> 21
> >  > commits are all over the place]
> >  >
> >  > If you are able to migrate to this branch - its best to delete
> > the old
> >  > one [i.e origin/mark/gamg-fix-viennacl-rebased]
> >  >
> >  > Satish
> >  >
> >  > On Fri, 2 Aug 2019, Mark Adams via petsc-dev wrote:
> >  >
> >  >> I have been cherry-picking, etc, branch
> > mark/gamg-fix-viennacl-rebased and
> >  >> it is very messed up. Can someone please update this branch when
> > all the
> >  >> fixes are settled down? eg, I am seeing dozens of modified files
> > that I
> >  >> don't know anything about and I certainly don't want to put in a
> > PR for
> >  >> them.
> >  >>
> >  >> I also seem to lose my pinToCPU method for cuda matrices. I don't
> >  >> understand how that conflicted with anyone else but it did.
> >  >>
> >  >> Thanks,
> >  >> Mark
> >  >>
> >  >
> >
>

Re: [petsc-dev] Is master broken?

[Mark Adams via petsc-dev]

>
>
>> several issues:
>>
>>
>> https://bitbucket.org/petsc/petsc/pull-requests/1954/cuda-fixes-to-pinning-onto-cpu/diff
>>
>>
> These links are dead.
>

I found one issue with not protecting the pinnedtocpu member variable in
Mat and Vec. Will fix asap.

Re: [petsc-dev] Is master broken?

[Balay, Satish via petsc-dev]

I don't really understand the workflow here [with merged branches (to
master) and cherry-picking other stuff]

I've attempted to rebase this branch against latest master. The result is at:

mark/gamg-fix-viennacl-rebased-v3

There were too many merge conflicts that I had to resolve. Its
possible I made mistakes here.

So if this branch is what you need - its best to check each commit
here and verify the changes - before using them.

As Karl suggests - its probably best to pick commits that you need and
fix them [if they have bad code - from merge conflict resolution]

[and some of them can be collapsed]

Satish

--

balay@sb /home/balay/petsc (mark/gamg-fix-viennacl-rebased-v3=)
$ git log --oneline master..
dee2b8b21b (HEAD -> mark/gamg-fix-viennacl-rebased-v3, 
origin/mark/gamg-fix-viennacl-rebased-v3, mark/gamg-fix-viennacl-rebased-v2) 
protected pinnedtocpu
03d489bde4 1) When detecting version info handle blanks introducted by 
preprocessor, error if needed version cannot be detected
056432fa93 Use outputPreprocess instead of preprocess since it prints source to 
log
94885b4f80 add back code missing from rebaseing over latest master?
2a748c2fa1 fixed compile errors
5e3edcb81f added check in Inode to skip GPUs for not Factor
fc14e5b821 removed comment
57224a7035 fixed up pinning CUDA to CPUs
346e0b6564 changed reduction logic a little and cleaned up format
3a282b2d9d added guards for empty process solves in CUDA
af6bc10070 same bug fix
0365c3a97f try a fix
9c37fbf3e7 bug fix
fd2e5db618 added cude test to ex56
72f2ad35b3 fixed bug with CUDA transpose mat-vec
ab62ce3476 remove vecset as per Barry
c26191aaa4 use non-collective VecSet
12042c4bfa removing ViennaCL fix to GAMG
3c46958f6d fix bug with empty processor
8bcb2d50b7 fixed MPI lock from call to collective method
54cfeb1831 added missing settypes
9508265e8e adding support for MatTranspose
e5a6000419 adding fix for ViennaCL in MG


On Mon, 12 Aug 2019, Karl Rupp via petsc-dev wrote:

> Hi Mark,
> 
> most of the CUDA-related fixes from your PR are now in master. Thank you!
> 
> The pinning of GPU-matrices to CPUs is not in master because it had several
> issues:
> 
> https://bitbucket.org/petsc/petsc/pull-requests/1954/cuda-fixes-to-pinning-onto-cpu/diff
> 
> The ViennaCL-related changes in mark/gamg-fix-viennacl-rebased can be safely
> discarded as the new GPU wrapper will come in place over the next days. ex56
> has not been pulled over as it's not running properly on GPUs yet (the pinning
> in your branch effectively turned GPU matrices into normal PETSc matrices,
> effectively running (almost) everything on the CPU again)
> 
> So at this point I recommend to start a new branch off master and manually
> transfer over any bits from the pinning that you want to keep.
> 
> Best regards,
> Karli
> 
> 
> On 8/3/19 8:47 PM, Mark Adams wrote:
> > Karl,
> > Did you want me to do anything at this point? (on vacation this week) I will
> > verify that master is all fixed if you get all my stuff integrated when I
> > get back to work in a week.
> > Thanks,
> > Mark
> > 
> > On Sat, Aug 3, 2019 at 10:50 AM Karl Rupp  > > wrote:
> > 
> > If you ignore the initial ViennaCL-related commits and check against
> > current master (that just received cherry-picked updates from your PR),
> > then there are really only a few commits left that are not yet
> > integrated.
> > 
> > (I'll extract two more PRs on Monday, so master will soon have your
> > fixes in.)
> > 
> > Best regards,
> > Karli
> > 
> > 
> > On 8/3/19 5:21 AM, Balay, Satish wrote:
> >  > I've attempted to rebase this branch over latest master - and pushed
> >  > my changes to branch mark/gamg-fix-viennacl-rebased-v2
> >  >
> >  > You might want to check each of your commits in this branch to see if
> >  > they are ok. I had to add one extra commit - to make it match 'merge
> >  > of mark/gamg-fix-viennacl-rebased and master'.
> >  >
> >  > This branch has 21 commits. I think its best if you can collapse them
> >  > into reasonable chunks of changes. [presumably a single commit
> > for all
> >  > the changes is not the correct thing here. But the current set of 21
> >  > commits are all over the place]
> >  >
> >  > If you are able to migrate to this branch - its best to delete
> > the old
> >  > one [i.e origin/mark/gamg-fix-viennacl-rebased]
> >  >
> >  > Satish
> >  >
> >  > On Fri, 2 Aug 2019, Mark Adams via petsc-dev wrote:
> >  >
> >  >> I have been cherry-picking, etc, branch
> > mark/gamg-fix-viennacl-rebased and
> >  >> it is very messed up. Can someone please update this branch when
> > all the
> >  >> fixes are settled down? eg, I am seeing dozens of modified files
> > that I
> >  >> don't know anything about and I certainly don't want to put in a
> > PR for
> >  >> them.
> >  >>
> >  >> I also seem to l

Re: [petsc-dev] Is master broken?

[Karl Rupp via petsc-dev]

Hi,

to clarify, these commits are already in master (either directly, or in 
a condensed way in the course of different pull requests, including 
additional fixes of memory leaks and output files):


> 5e3edcb81f added check in Inode to skip GPUs for not Factor
> 346e0b6564 changed reduction logic a little and cleaned up format
> 3a282b2d9d added guards for empty process solves in CUDA
> af6bc10070 same bug fix
> 0365c3a97f try a fix
> 9c37fbf3e7 bug fix
> 72f2ad35b3 fixed bug with CUDA transpose mat-vec
> ab62ce3476 remove vecset as per Barry
> c26191aaa4 use non-collective VecSet
> 8bcb2d50b7 fixed MPI lock from call to collective method
> 3c46958f6d fix bug with empty processor
> 54cfeb1831 added missing settypes

this commit is not in master because I couldn't get the example to run:
> fd2e5db618 added cude test to ex56

these commits are ViennaCL-specific and don't need further consideration 
(the wrapper will take care of it):


> 12042c4bfa removing ViennaCL fix to GAMG
> 9508265e8e adding support for MatTranspose
> e5a6000419 adding fix for ViennaCL in MG


The *only* commit with functionality that is not in master is:

> 57224a7035 fixed up pinning CUDA to CPUs

for which PR #1954 holds the discussion of why this commit has 
problematic parts.


I've opened a pull request for Barry's fixes to the builds on Summit in 
PR #1963. If there are no surprises, this PR will be in master by the 
end of the day.


**Overall**: Please start freshly off master (and merge PR 1963 if 
needed). Then, have a look at the pinning-to-CPUs-commit and decide 
whether it needs to be reworked as for what is discussed in PR #1954. Do 
not try to rebase

 mark/gamg-fix-viennacl-rebased
or any of its offsprings any further, as this will only cause headaches 
and create conflicts with code that has already been fixed.


Best regards,
Karli



On 8/12/19 5:23 PM, Balay, Satish wrote:

I don't really understand the workflow here [with merged branches (to
master) and cherry-picking other stuff]

I've attempted to rebase this branch against latest master. The result is at:

mark/gamg-fix-viennacl-rebased-v3

There were too many merge conflicts that I had to resolve. Its
possible I made mistakes here.

So if this branch is what you need - its best to check each commit
here and verify the changes - before using them.

As Karl suggests - its probably best to pick commits that you need and
fix them [if they have bad code - from merge conflict resolution]

[and some of them can be collapsed]

Satish

--

balay@sb /home/balay/petsc (mark/gamg-fix-viennacl-rebased-v3=)
$ git log --oneline master..
dee2b8b21b (HEAD -> mark/gamg-fix-viennacl-rebased-v3, 
origin/mark/gamg-fix-viennacl-rebased-v3, mark/gamg-fix-viennacl-rebased-v2) 
protected pinnedtocpu
03d489bde4 1) When detecting version info handle blanks introducted by 
preprocessor, error if needed version cannot be detected
056432fa93 Use outputPreprocess instead of preprocess since it prints source to 
log
94885b4f80 add back code missing from rebaseing over latest master?
2a748c2fa1 fixed compile errors
5e3edcb81f added check in Inode to skip GPUs for not Factor
fc14e5b821 removed comment
57224a7035 fixed up pinning CUDA to CPUs
346e0b6564 changed reduction logic a little and cleaned up format
3a282b2d9d added guards for empty process solves in CUDA
af6bc10070 same bug fix
0365c3a97f try a fix
9c37fbf3e7 bug fix
fd2e5db618 added cude test to ex56
72f2ad35b3 fixed bug with CUDA transpose mat-vec
ab62ce3476 remove vecset as per Barry
c26191aaa4 use non-collective VecSet
12042c4bfa removing ViennaCL fix to GAMG
3c46958f6d fix bug with empty processor
8bcb2d50b7 fixed MPI lock from call to collective method
54cfeb1831 added missing settypes
9508265e8e adding support for MatTranspose
e5a6000419 adding fix for ViennaCL in MG


On Mon, 12 Aug 2019, Karl Rupp via petsc-dev wrote:


Hi Mark,

most of the CUDA-related fixes from your PR are now in master. Thank you!

The pinning of GPU-matrices to CPUs is not in master because it had several
issues:

https://bitbucket.org/petsc/petsc/pull-requests/1954/cuda-fixes-to-pinning-onto-cpu/diff

The ViennaCL-related changes in mark/gamg-fix-viennacl-rebased can be safely
discarded as the new GPU wrapper will come in place over the next days. ex56
has not been pulled over as it's not running properly on GPUs yet (the pinning
in your branch effectively turned GPU matrices into normal PETSc matrices,
effectively running (almost) everything on the CPU again)

So at this point I recommend to start a new branch off master and manually
transfer over any bits from the pinning that you want to keep.

Best regards,
Karli


On 8/3/19 8:47 PM, Mark Adams wrote:

Karl,
Did you want me to do anything at this point? (on vacation this week) I will
verify that master is all fixed if you get all my stuff integrated when I
get back to work in a week.
Thanks,
Mark

On Sat, Aug 3, 2019 at 10:50 AM Karl Rupp mailto:r...@iue.tuwien.ac.at>> wrote:

 If you ign